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Executive Summary 


For most workers, a high school diploma or credential is not sufficient to succeed in the modern 
economy. Earnings have stagnated for those whose highest level of education is a high school degree. As 
of 2017, workers with only a high school degree earned 44 percent less than workers with some college 
education or four-year degrees.* 


Moreover, even adults who have high school credentials frequently come to college underprepared, 
with below-college-level skills. By one estimate, community colleges referred approximately 60 percent 
of first-time enrolling students to at least one developmental math class and about 33 percent to at 
least one developmental reading class (Bailey, Jeong, and Cho 2010). 


The Accelerating Opportunity (AO) initiative was developed to address these issues by giving 
underprepared students and adults without high school credentials an opportunity to enroll in 
integrated career pathway programs at community and technical colleges. AO was based on 
Washington State’s Integrated Basic Education and Skills Training (I-BEST) model and lessons from the 
Breaking Through initiative.2 AO allowed students scoring in the 6th- through 12th-grade National 
Reporting System (NRS) educational functioning levels to enter career and technical education (CTE) 
courses concurrently with high school equivalency (HSE) completion programs through adult education 
or other skill-building courses. The pathways offered efficient course offerings with paths to multiple 
stackable, industry-recognized credentials within about 12 credit hours. To promote students’ 
postsecondary success, colleges participating in AO provided team teaching in at least 25 percent of 
their classes, where a CTE instructor worked alongside an adult education instructor in the classroom, 
as well as contextualized instruction, accelerated learning, supportive navigation services, and 
connections with employers and workforce agencies to help students complete their coursework and 
transition from AO pathways to the workforce. 


With support from several foundations, notably the Bill & Melinda Gates Foundation, Jobs for the 
Future (JFF) partnered with national technical assistance providers to administer AO and provide 
technical assistance to seven participating states.? JFF also specified program elements and 
performance outcomes. The Urban Institute and its partners at the Aspen Institute and the George 
Washington University led a rigorous evaluation of AO in four states to inform policymakers and 
practitioners on the model’s potential to improve postsecondary education and employment outcomes 
for adults with low basic skills. 


Impact Evaluation 


This final report from the AO impact evaluation presents estimates of how AO career pathway 
programs affected the educational and employment outcomes of participants in Illinois, Kansas, 
Kentucky, and Louisiana. The impact analysis examined four key educational outcomes of AO: the 
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number of credits earned, earning at least 12 academic credits, earning any credential offered by a 
community college, and the number of credentials earned. The analysis also estimated impacts on two 
labor-market outcomes: the probability of being employed after enrollment (measured each quarter) 
and the quarterly earnings of AO participants. These outcomes reflect the principal goal in AO’s theory 
of change: to improve the educational and employment trajectories of underprepared adult learners 
and thereby increase their employment and earnings (see appendix A). 


Using matched comparison groups in each state, the research team conducted a rigorous 
propensity score matching analysis. This approach matched AO students with a comparison group of 
non-AO students that were similar in almost every measurable way (including prior labor-market 
activity, test scores, education history, demographics, and timing of college enrollment) except for their 
participation in AO. This comparison group provides an estimate of how AO students would have fared 
in the absence of AO. The difference between the outcomes of AO participants and the outcomes of the 
comparison group provides an estimate of how much better students fared due to AO enrollment: the 
impact of AO on the outcomes of interest. The research team tracked students for 2 to 11 semesters 
after AO enrollment to measure education outcomes and for 3 to 12 quarters after AO enrollment to 
measure labor-market outcomes, depending on the availability of data. Earlier cohorts of AO students 
have longer observed follow-up periods than later cohorts. 


In Illinois and Louisiana, colleges recruited overwhelmingly from the adult education population, as 
was envisioned when the AO initiative was developed. In Kansas and Kentucky, colleges recruited both 
from adult education and from an additional college source: current CTE students in Kansas and 
developmental education students in Kentucky. The matching analysis was completed separately by 
recruitment source to account for unmeasured differences among those groups, and the results are 
reported for each subgroup of students and overall for the state. 


This impact analysis presents results for a total of 4,361 students who appeared in college records 
as having enrolled in at least one credit-bearing course, were not English-language learners, had a valid 
Social Security Number, and were retained in the matching procedures utilized for the analysis. This 
represents 63 percent of the 6,946 students flagged as AO participants in the states’ administrative 
records. Many students were excluded from the analysis because they did not take a credit-bearing 
course and therefore did not have college records, often because of enrollment in noncredit AO 
programs, which were especially common in Louisiana. 


Findings 


Based on survey data, Anderson and colleagues (2016) reported that colleges counted a total of 8,287 
students enrolled in AO in the four states in the first three years of the initiative, which covered 
calendar years 2012 through 2014 in Illinois, Kansas, and Kentucky and fall 2012 through summer 
2015 in Louisiana. The survey results indicated that AO students enrolled in 154 integrated career 
pathways, primarily in health care and manufacturing. 
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The college administrative records included over 4,500 AO students with data. For these students, 
AO colleges awarded over 79,000 credits and nearly 6,800 credentials, producing an average of over 17 
credits and 1.5 credentials per AO participant. Examples of credentials included college-awarded 
certificates for certified nurse aides (CNAs), computer numerical control (CNC) operators, welding 
technicians, and automotive technicians. One key outcome that JFF specified for each state was that 
AO participants earn an aggregate of 3,600 occupational credentials through AO pathways. 


Based on quasi-experimental propensity score matching analysis on the 4,361 students retained in 
the sample, the evaluation found that AO had a positive impact on the number of college-awarded 
credentials earned by almost all students. In most cases, AO students earned more credentials while 
taking fewer credits, possibly indicating more efficient course-taking and accelerated learning. Figure 1 
and figure 2 summarize the credential attainment gains by state. The percent increases, noted above 
each bar, were much larger in states where the matched comparsion group had low credential 
attainment, particularly Louisiana. In Louisiana, the 20 percentage-point impact of AO on the likelihood 
of earning any credential relative to the comparsion group’s average of 3 percent likelihood results ina 
622 percent increase. Because the comparison group earned zero credentials on average, the percent 
increase for number of credentials earned is undefinable. In Kansas, the comparsion group had a very 
high rate of credential attainment. Though they had similar absolute gains to Illinois and Kentucky (a 13 
percentage-point gain in likelihood of earning a credential and a 0.6 credential gain in number of 
credentials earned), the relative increases were smaller. 


FIGURE 1 


Likelihood that Accelerating Opportunity Students Earned Any Credential, Relative to Matched 
Comparison Group, by State 


= Comparison m= AO Impact 


19% increase 


35% increase 


133% increase 


622% increase 


Note: This summary figure does not separate the Kansas and Kentucky results by recruitment source. 
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FIGURE 2 


Average Number of Credentials Earned by Accelerating Opportunity Students, Relative to Matched 
Comparison Group, by State 


m= Comparison = AO Impact 


40% 
increase 


78% 
increase 


25% 
increase 


Undefined 
increase 

Oj 43° 

IL KS KY LA 


Note: This summary figure does not separate the Kansas and Kentucky results by recruitment source. 


The positive outcomes for credential attainment are notable, though they did not always translate into 
labor market gains in the observed timeframe. AO had strong and sustained positive impacts on 
earnings for two subgroups: AO students recruited from adult education in Kentucky and AO students 
recruited from CTE in Kansas. Adult education students from Illinois, Kansas, and Louisiana and 
developmental education students from Kentucky did not achieve positive, statistically significant, or 
enduring gains in earnings during the follow-up period. Table 1 highlights the overall patterns of impact 
in each state. 


The state-by-state results summarize the levels, impact, and percent change in key outcomes and 
the levels of those outcomes for the treatment and comparison groups. The percent change is measured 
relative to the comparison group, so comparison groups with very low levels have higher percent 
changes for the same AO impact. Full results appear in the report body and in appendix C. 
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TABLE 1 


Summary of Accelerating Opportunity Impact Findings from 


Quasi-Experimental Analysis, by Recruitment Source 


Illinois Kansas Kentucky Louisiana 

Adult ed. | Adult ed. CTE Adulted. Dev.ed. | Adult ed. 
Education 
Credential attainment 4 - 4p a 4 + 
Credit attainment (persistence) 1) - - + + - 
Labor Market 
Short-term employment - ° 4 - 4 Fe 
Mid-term employment + e + tr + ) 
Long-term employment + N/A N/A 4 a ) 
Short-term earnings - + + - ) af 
Mid-term earnings sf e 4 + ° % 
Long-term earnings ° N/A N/A + 0) = 
Accelerating Opportunity Sample Size 867 459 1,239 122 1,234 440 


Notes: 


+ = significant positive impacts, e = no significant impacts, - = significant negative impacts, = both positive and negative significant 


impacts; short-term = quarters 1-3, mid-term = quarters 4-8, long-term = quarter 9 through final observations; N/A = not 


applicable. Kansas does not have long-term impacts because students are only observed for eight quarters following enrollment. 


The short-term impacts largely reflect the in-program period, when it may be reasonable to expect negative labor-market impacts 


because many in training may not be working. 


Ilinois 


Key AO recruitment source: Primarily adult education students. 


=" AOstudents earned 25 percent more credentials than the comparison group and were 35 


percent more likely to earn any credential than the comparison group. As shown in figure 1 


and figure 2, the comparison group’s credential attainment rate was fairly high, but the average 


number of credentials earned was quite low. 


= AOstudents earned fewer college credits, suggesting acceleration. They were more likely to 


persist beyond 12 credits. 


= —AOstudents had somewhat higher employment rates than the comparison group beginning in 


the 3rd quarter, peaking at a 15 percent gain over the comparison group rate of 54 percent in 


the 12th quarter after AO enrollment (impact of 8 percentage points). 


= AOstudents had medium-term earnings gains in the fifth quarter after enrollment of up to 14 


percent over the comparison group average earnings of $2,035 (impact of $293), though these 


positive impacts faded out by the sixth quarter after enrollment. 
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Kansas 


Key AO recruitment source: Primarily CTE students, but also a relatively low-barrier subset of adult 
education students. 


= AOstudents from both adult education and the CTE recruitment earned more credentials than 
their matched comparison groups. AO students from CTE earned 57 percent more credentials 
than the comparison group. The gain for AO students from adult education was positive but not 
significantly different from zero. AO students recruited from CTE were also more likely to 
earn any credential than their matched comparison group, but AO adult education students 
were not. 


= AOstudents from both the adult education and the CTE groups attained more credentials while 
taking fewer college credits, suggesting acceleration. 


= The CTE group experienced large, positive, and persistent impacts on employment and 
earnings. The employment gains in the eighth quarter after enrollment were 33 percent higher 
than the comparison group average earnings of $3,606 (impact of $1,188). 


= The adult education group experienced close to zero impacts on both labor-market 
measures. However, it should be noted that this was a particularly low-barrier subset of the 
adult education population that would likely have had strong labor-market outcomes in the 
absence of AO (as demonstrated by the comparison group). 


Kentucky 


Key AO recruitment source: Primarily developmental education students, but also adult education 
students. 


= AOstudents recruited from both adult education and developmental education experienced 
relatively large increases in the number of credentials earned and the probability of earning 
any credential. AO students recruited from adult education were over six times more likely to 
earn any credential. AO students recruited from developmental education earned 80 percent 
more credentials and were more than twice as likely to earn any credential. These large gains 
reflect the low levels of credential attainment among the comparison groups. 


= AOstudents recruited from both adult education and developmental education also earned 
more credits and were more likely to earn 12 credits than their respective comparison groups. 
On average, AO adult education students earned additional credits equating to about two more 
classes, and developmental education students earned credit equating to about 1.5 more 
classes. 
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= Students from both recruitment sources experienced positive impacts on employment rates, 
with larger impacts among AO students recruited from adult education. Only AO students 
from adult education experienced earnings gains over the comparison group, reaching a gain 
of 43 percent over the average comparison group earnings of $1,984 in the ninth quarter after 
enrollment (impact of $855). For AO students from adult education, substantial gains persisted 
through the end of the observed period. Modest but significant positive gains appeared only in 
the 12th observed quarter for students recruited from developmental education. 


Louisiana 


Key AO recruitment source: Primarily adult education students; program also offered many noncredit 
pathways that are excluded from the analysis. 


= AOstudents in Louisiana saw gains in the number of credentials earned. They also saw large 
gains in the likelihood of earning any credential, with AO students 20 percentage points more 
likely to earn a credential than the matched comparison group, a 622-percent increase. 


a Meanwhile, AO students earned fewer credits, equating to more than one fewer classes taken. 
They were less likely to persist for 12 credits, suggesting acceleration. 


= AOstudents saw gains in employment and earnings through the sixth quarter after 
enrollment of up to 38 percent more than the comparison group average of $1,868 (impact of 
$710), after which they fell behind the comparison group on earnings, displaying negative 
impacts. That may have occurred because of the various similar programs and services 
available to the comparison group in Louisiana during AO implementation, which would 
weaken the distinct effect of the AO treatment over what was otherwise available. 


Overall, AO helped participants with low academic skills earn more credentials from community 
college programs than similar non-AO students. AO students often increased their credential 
attainment while taking fewer credits. Depending on the value of the certificates, this pattern may 
represent a cost savings in terms of tuition and time dedicated to earning credentials. Labor-market 
gains for AO participants were mixed. Most AO students were not able to translate added certificates 
into consistent employment and earnings gains in the observed period. For Kansas CTE students and 
Kentucky adult education students, however, the earnings impacts were positive and persistent. 


Limitations 


The analysis did not take into account potential benefits to students, such as self-esteem, family 
stability, health, or other aspects of personal well-being. In addition, many states and colleges used AO 
as an opportunity to implement systemic changes to improve opportunities for adult education and 
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other low-skilled students. Elements of systems change are not captured in this analysis, though some 
aspects are described in the implementation report (Anderson et al. 2016). 


In addition, the AO population is a subset of students that may not represent the broader 
populations in their states of those in adult education, CTE, and developmental education. For example, 
the Kansas AO adult education group appears to be particularly advantaged as measured by 
educational attainment, prior employment, NRS educational functioning levels, and the earnings of the 
comparison group. The unique characteristics of AO students may limit the generalizability of the 
findings across larger groups. 


Other analysis limitations are discussed in the Methodology section of the report. 


Policy Implications of AO Impact Results 


AO is a promising approach to help low-skilled adults attain more credentials, potentially more quickly 
than they would otherwise. But AO alone may be insufficient for generating consistent, positive effects 
on earnings. Policymakers and administrators considering replication of AO should look closely at how 
best to translate increases in credential attainment into long-term earnings gains. The evaluation’s 
implementation research (Anderson et al. 2016) indicates that strengthening linkages with employers 
may be a critical component that was not fully developed in the early implementation period. In 
addition, further development of the model to improve labor-market outcomes might focus on helping 
students advance their careers beyond the jobs associated with entry-level credentials. 
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Overview of Accelerating 
Opportunity and Prior Research 


Accelerating Opportunity (AO) offered underprepared adult learners the opportunity 
to enroll in career and technical education (CTE) pathways at community and technical 
colleges even if they did not possess a high school diploma or high school equivalency 
(HSE), collectively termed a high school credential, or had low test scores that otherwise 
would have limited their access to for-credit CTE courses. To increase the success of 
these students in their CTE courses, AO enhanced the supports for participating 
students by using team teaching, contextualized instruction, and career navigation. At 
the same time, students could complete their HSE through adult education or remediate 
low basic skills. The initiative also supported systems changes at the college and state 
level. These changes encouraged more sustainable and career-focused career pathways 
by shifting culture to make the colleges and CTE programs more welcoming to students 
with low initial test scores or without high school credentials and by altering policy 
conditions to help fund and support underprepared students. The Urban Institute and 
its partners conducted a mixed-methods evaluation to document AO implementation, 
estimate its impacts on participants’ education and employment outcomes, and assess 
whether the effort yielded greater benefits than costs over time. This report presents 
the results of the analysis of AO impacts on students’ education and labor-market 
outcomes. 

Launched in 2011, AO aimed to transform how states and community and technical colleges train 
and educate students with low basic skills. Past research had found that many students with low scores 
on academic tests spend sometimes considerable time in remedial or developmental education classes 
and then often do not complete all occupational courses that are required to attain a credential. The AO 
model offered a new approach based on the Integrated Basic Education and Skills Training (I-BEST) 
program developed and operated in Washington State. Instead of requiring students with weak 
academic skills to complete preparatory courses before entering CTE courses, the AO model moved 
students directly into CTE courses at community and technical college while adding a second teacher 
who assisted students with basic skills reinforcement in reading and math. The critical team teaching 


component involved adult education and CTE instructors working together in the same classroom for 
about 25 percent of the total class time in the pathway. In addition to team teaching, AO incorporated 


comprehensive student support services, accelerated learning, and alignment between basic skills 
instruction and substantive technical concepts. 


This report presents findings from a quasi-experimental impact analysis of the effect of AO on 
participating students’ educational outcomes, employment, and earnings.* This work complements final 
reports from the Urban Institute on AO implementation (Anderson et al. 2016) and cost-benefit 
analysis (Kuehn et al., forthcoming). 


The AO evaluation began in 2012 and focused on programming through the end of 2014. Because 
states varied in the operational aspects of the programs implemented within the AO model, the study 
examines impacts separately for each of the four participating states: Illinois, Kansas, Kentucky, and 
Louisiana. AO generally had positive effects on the attainment of CTE credentials awarded by colleges 
but had mixed effects on college persistence. It also had positive effects on postprogram employment 
and some limited positive effects on postprogram earnings that persisted over time in a couple of cases, 
but faded in most other instances. The effects varied across states and student populations, suggesting 
that AO exerted different impacts depending on the population served, local economic context, and 
implementation conditions. 


The Rationale for Accelerating Opportunity 


AO aimed to address a major challenge faced in the United States: there are too many adults with low 

basic skills and few prospects to earn a decent living. About one in six adult Americans (36 million) has 

low literacy levels, and one in three (18 million) has low numeracy levels (OECD 2013; US Department 
of Education 2015). 


Adults with low basic skills may or may not have high school credentials. Eleven percent of adults 
lack a high school diploma or HSE, such as a General Education Development (GED) credential. The 
average unemployment rate in 2015 for adults ages 25 to 64 without high school credentials was 78 
percent higher than the unemployment rate among those with some college education or greater.? As of 
2017, workers with only a high school degree earned 44 percent less than workers with some college or 
four-year degrees.° 


Adult education programs, operated by community and technical colleges, school districts, and 
community-based organizations, help adults obtain a secondary school credential, such as an HSE 
certificate or adult high school diploma, or help them improve English-language skills. However, such 
programs typically have few links to postsecondary education or advanced training that yield 
recognized occupational credentials necessary for well-paying jobs. By one estimate, only 3 to 6 percent 
of adult education students transition to postsecondary programs and earn any type of certificate (US 
Department of Education 2013). Thus, few adult education students ever enroll in, much less complete, 
postsecondary education or advanced training. 


[Zz] IMPACT FINDINGS FOR ACCELERATING OPPORTUNITY (AO) 


Adults who do have high school credentials may be low-skilled as well. Many high school graduates 
are underprepared for postsecondary education and are placed in developmental education classes 
when they enroll in college. By one estimate, community colleges referred approximately three-fifths of 
first-time enrolling students to at least one developmental math class and referred one-third to at least 
one developmental reading class (Bailey, Jeong, and Cho 2010). Thus, adults with high school 
credentials often also require remediation. 


About three-quarters of the fastest-growing jobs in the next decade will require a high school 
credential and some postsecondary education or advanced training.’ Further, as the economy shifts 
with technological advances, falling oil prices, and other macroeconomic changes, workers will 
increasingly need higher levels of literacy and numeracy. Innovative approaches to coenrollment and 
career pathways, such as AO, may help adults access the education necessary for labor-market success. 


The Accelerating Opportunity Model 


The AO initiative was a state-led effort funded by grants from several foundations, particularly th e Bill 
& Melinda Gates Foundation, and administered by Jobs for the Future (JFF).® State-level teams of 
community college oversight offices and partners guided and funded a minimum of eight participating 
community and technical colleges to develop or modify college programs to emphasize career pathways 
for in-demand occupations and make them accessible to AO participants. Career pathways are 
sequenced education and training programs for gaining occupational knowledge in in-demand fields; 
they allow students to quickly earn an initial credential and build on it with additional related 
credentials later (Clagett and Uhalde 2012; CLASP 2013; Fein 2012). AO career pathways were 
designed to make participation and completion manageable for low-skilled individuals with family and 
work commitments and to help students develop marketable occupational skills. In AO, the initial phase 
of the pathway typically consisted of approximately 12 credits and lasted one academic year or less. 
This requirement for AO career pathways was based on “tipping point” research, which suggests that 
once a student completes the equivalent of one full-time semester of college course work, they are 
more likely to pursue further education and training (Prince 2009).? During this phase, a student earned 
a college-awarded credential or set of credentials that would be meaningful for employment through 
CTE programs. Subsequent steps on the pathway allowed students to earn additional credentials and 
degrees to help them qualify for mid- to high-skilled occupations that paid good wages. 


Beyond career pathways, other key programmatic components of AO included integrated 
instruction, where both basic skills and CTE instructors taught in the same classroom with at least 25 
percent overlap (known as the “team teaching approach’); a focus on comprehensive student support 
and navigation services; contextualization of instruction; accelerated learning; and labor-market 
connections. JFF and its partners derived those approaches from Washington State’s |-BEST model and 
lessons from the Breaking Through initiative.*© A quasi-experimental impact study of the I-BEST model 
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found positive effects on students’ college credits and credentials earned but no detectible effects on 
persistence in college and labor-market outcomes (Zeidenberg, Cho, and Jenkins 2010). 


These initial findings spurred replication, adaptation, and scale-up of those approaches to new 
states through AO in an effort to both improve underprepared students’ academic outcomes and affect 
their labor-market success through several enhanced elements. AO’s enhanced design included policy 
changes, partnerships, and culture shifts at the college and state levels to institutionalize the model. 
This means that JFF expected states and colleges to create and maintain a system of support for low- 
skill students to access, be accepted in, succeed in postsecondary education, and experience labor- 
market payoffs. The states involved in the evaluation (Illinois, Kansas, Kentucky, and Louisiana) agreed 
to adhere to the AO model and the required program elements. Appendix A summarizes the core design 
elements of the AO model and presents the theory of change, which specifies two- and four-year 
outcomes for the system and long-term goals for the students.*? (Note that all appendices appear ina 
separate document.) 


JFF’s overall goal for the initiative was for each participating state to award at least 3,600 
credentials within the grant period. Participating colleges were required to target recruitment efforts 
toward students who are within National Reporting System educational functioning levels (NRS 
functioning levels) 4-6 (6th- to 12th-grade level) on math, reading, or writing or NRS functioning levels 
5-6 (high intermediate to advanced) in English-language skills for English-language learners. Eligible 
students may or may not have had high school diplomas or HSE credentials, though the initiative was 
designed to focus on adult education students without an HSE or with low English-language skills. Shifts 
in the target population as states sought to scale over the course of the initiative and the implications 
for the evaluation are discussed below. 


The AO Evaluation 


The AO evaluation, conducted by the Urban Institute and its partners, the Aspen Institute and George 
Washington University, is a comprehensive assessment of the initiative. The goal is to generate valuable 
evidence for the field and inform public policy on new approaches to serving the education and 
workforce needs of adults with low basic skills. 


The evaluation consists of three major components: 


= Implementation study: A qualitative study of how AO integrated pathways were undertaken 
by the states and colleges, scaled, and potentially sustained, as well as an analysis of how well 
the states and colleges implemented the AO model. Anderson et al. (2016) report the final 
implementation findings. 


= Impact study: A quasi-experimental analysis designed to measure the effectiveness of the AO 
model based on its impact on the education and labor-market outcomes of AO participants, 
comparing them with similar students who did not participate in AO. 
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=" Cost-benefit analysis: A comparison of the costs and benefits for states, colleges, and students 
engaged in the AO initiative. The final report is forthcoming (Kuehn et al., forthcoming). 


This report provides the results of the final impact study, with medium- and long-term impacts on 
students’ education and employment outcomes for all cohorts of AO students. This report intends to 
answer the following research questions: 


= Who were the AO students and how did they differ across states? 
=" _Howdid colleges’ AO pathways and recruitment efforts shape the mix of students in AO? 
=" Howdid the characteristics of AO students change over time? 


5 Did AO students earn more credentials and credits than similar individuals who did not enroll in 
AO? 


= Were some groups of AO students more successful than others in their educational outcomes? 


= Did AO students achieve higher employment and earnings levels than similar individuals who 
did not enroll in AO? 


= Were some groups of AO students more successful than others in their labor-market 
outcomes? 


The impact evaluation examined students who enrolled in the first three years of the initiative, 
which included cohorts enrolling from calendar years 2012 to 2014 in Illinois, Kansas, and Kentucky 
and cohorts enrolling through summer 2015 in Louisiana. The research team used propensity score 
matching, a quasi-experimental statistical method that matched treatment and comparison group 
members on observed characteristics to estimate the impacts of AO on educational and employment 
outcomes, as described in the Methodology chapter below.’? In the design phase of the evaluation, the 
team considered using regression discontinuity design, given the eligibility cut-offs using test scores, 
but inconsistencies in implementation of eligibility requirements made this method infeasible. Data for 
the impact evaluation came from state college and earnings administrative records, and measures were 
therefore limited to what could be observed in these data sources. The implementation findings, 
derived from site visits, interviews, student focus groups, annual college surveys, and two rounds of 
student surveys, informed the research team’s interpretation of the impact results. 


AO Implementation Findings 


The AO implementation study has been important for shaping the impact study design and for 
interpreting of the results.*? Key findings from the AO implementation report are as follows: 


The composition of participating colleges and pathways changed over time as states and colleges 
scaled the initiative. States were required by the terms of the grant to engage at least eight colleges in 
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AO. Thirty-four colleges were selected at the start of the initiative across the four states.’* Kansas and 
Louisiana began the implementation period with nine colleges, and Illinois and Kentucky started with 
eight. Over time, the number of AO colleges increased in all states except Louisiana. Kentucky scaled 
AO up to all 16 colleges in the state by the end of the third year of implementation. Illinois added four 
more colleges each year, and Kansas expanded to an additional two colleges throughout 
implementation (though one Kansas college discontinued participation in the second year). Further, 
colleges made changes to the number and types of pathways offered, growing from 89 to 154 recorded 
pathways; colleges also changed the nature of team teaching and supportive services in each pathway 
area. As states and colleges gained experience over the three years, the initiative evolved and grew. The 
research team has included all participating colleges in the impact study, though the cost-benefit 
analysis (forthcoming) is limited to colleges that participated in AO for all three years and that provided 
complete cost data in each year. The section “Cohort Effects” contains impact results for only the 
colleges involved in the cost-benefit analysis by student cohort year. 


Early implementation of the AO pathways was challenging, but colleges became more strategic 
about the pathways offered. Early in the AO implementation period, colleges worked intensively to roll 
out pathways quickly and build the necessary infrastructure and supports to deliver integrated 
instruction, contextualized curriculum, and team teaching. When selecting the occupational areas that 
pathways would target, many colleges evaluated local labor-market data in addition to prioritizing 
institutional factors, such as building the support of CTE faculty for the AO model or modifying 
enrollment requirements for certain pathways. By the second and third years of the initiative, states 
and colleges had more experience with the AO model and could be more strategic about how pathway 
offerings could match the needs of both students and local employers. Manufacturing and health care 
remained the most common occupational areas, at 39 and 32 percent, respectively, for career 
pathways. With guidance from AO state teams, many colleges reemphasized labor-market demand in 
selecting pathways rather than institutional priorities and recalibrated some of their pathway offerings. 
Given that the majority of pathways and students fell within the manufacturing and health care sectors, 
the impact analysis categorizes AO students who are not in one of these two pathway areas in “other” 
sectors. 


The instructional methods used and the experience of the teaching teams varied over time. 
During the first year of the initiative, colleges prioritized training the first teams of CTE and adult 
education faculty in integrated instruction, including contextualized instruction and team teaching 
methods. By the end of the first year, all colleges in the four states had begun team teaching, 
implementing diverse styles across colleges and classrooms. As with pathway selection, the initial 
decision on how and in which courses to implement team teaching depended greatly on the support of 
CTE faculty and staff. In the second and third years, colleges continued to bring new faculty into the 
initiative and train them in team teaching and integrated instructional approaches. 


Colleges and teaching pairs implemented team teaching in different styles, ranging from an 
approach in which the CTE and adult education instructors share teaching duties to an approach in 
which the CTE teacher is responsible for instruction while the adult education instructor circulates 
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around the classroom, monitoring student needs in real time. Variations in students’ team teaching 
experiences may explain some of the variation in impacts. It was not possible to track team teaching 
intensity or approach in the data used for the impact study. 


In Kansas and Kentucky, many colleges recruited individuals with high school diplomas or HSEs 
and struggled to recruit adult education students. One reason for this focus on students with high 
school credentials was that students without high school credentials lost access to federal Pell grants 
(known as the “ability-to-benefit” provision) to pay for program tuition soon after the initiative started 
in July 2012. The change in federal rules meant these students could no longer qualify for federal 
financial aid. The requirement meant that non-HSE students enrolling into most AO programs would 
need to pay for their own college tuition costs and work with AO college staff on strategies to finance 
their education. This federal policy change affected colleges’ efforts to recruit students lacking high 
school credentials, who typically are enrolled in adult education programs. As a result, many AO 
colleges in Kentucky and Kansas heavily recruited students who met the AO test score targets, meaning 
the students qualified for AO at the allowable skill level but already had high school credentials and thus 
could qualify for Pell grants. These students came from the colleges’ developmental education or CTE 
programs. Some stakeholders explained in interviews that they did not see focusing on students already 
in developmental education or CTE programs as an issue. They pointed out that AO could significantly 
improve education and earnings outcomes for students with and without high school credentials, giving 
them the ability to succeed in college courses through the provision of additional supports, particularly 
team teaching. Because colleges heavily recruited students outside of adult education in Kansas and 
Kentucky, the research team divided AO students by recruitment source in those two states and 
matched to comparison group students from the same source. This is discussed in more detail below. 


Although scores were a primary determinant of eligibility, colleges used several screening 
methods to enroll qualified students. When recruiting students for AO, colleges considered not only a 
student’s adult basic skills or English-language proficiency test scores but also their level of 
commitment to the program and potential barriers to completion, such as lack of transportation and 
funding issues. In order to meet program requirements and maximize the chance of student success, 
many colleges screened students based on criteria other than test scores, such as by requiring 
interviews, recommendations, background checks, or proof of citizenship. In the first year of AO, 85 
percent of colleges used some additional screening mechanism (Anderson et al. 2014). AO colleges had 
a mix of experiences in recruiting AO students in the first year, but many college coordinators expected 
that they would see increased enrollment numbers as awareness of the program grew and they could 
prepare more students to enter into pathways.’ The determination of eligibility based on factors 
beyond test scores complicates the impact study because the propensity score matching approach used 
in the impact analysis can only rely on characteristics that are observable in the data. If student 
motivation or personality characteristics are not well-captured by the available variables, then a 
mismatch (bias) will occur between the treatment and comparison groups. For this reason, the research 
team included a wide range of characteristics, including prior earnings and test scores, in the matching 
model. 
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Colleges engaged employer partners as a part of AO, but many faced challenges in creating and 
sustaining meaningful relationships. AO programs with strong connections with CTE departments 
often leveraged existing employer advisory boards to connect employers directly with students, create 
opportunities for work-based learning, or promote or support the AO program in other ways. About 35 
percent of AO students engaged in work-based learning over the three years, according to participating 
colleges. The percentage increased from 28 percent in the first year to 42 percent in the third year of 
AO. As those numbers suggest, the relationships between AO programs and employers often developed 
over time. Employer engagement was particularly weak in the early implementation period. Further, the 
level of employer engagement varied widely across states, colleges, and even pathway occupational 
areas throughout the implementation period. That variation may have had some bearing on students’ 
labor-market outcomes after completing the initial AO pathway and may be reflected in the modest 
employment and earnings impacts found in this report, which reflect the longest-term outcomes for the 
earliest AO cohorts. 


The remaining report chapters provide information on the characteristics of AO participants and 
the education and labor-market impacts of each state program. The discussion explains how known 
information about AO implementation can help explain the impacts on AO students’ educational and 
earnings outcomes. The appendices present detailed participant characteristics, the results of the 
analysis of AO impacts on education, and the labor-market outcomes of AO participants. 
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Methodology 


This impact analysis aims to answer the question, “What happened to AO students 
compared to what would have happened to these students in the absence of AO?” To 
estimate that alternative scenario (the counterfactual), this report uses a quasi- 
experimental technique called propensity score matching, an approach that identifies a 
comparison group of individuals similar to AO students but who were not exposed to 
AO. The propensity score is the predicted probability that any individual in the dataset 
could be an AO participant. Using the propensity score, the research team matched the 
students who participated in AO (the treatment group with students who have the same 
or similar propensity scores who did not have the opportunity to participate in AO (the 
comparison group). The comparison groups were drawn from the same recruitment 


source as AO students: adult education, developmental education, or CTE. 


The matching between treatment and comparison groups is at the individual level. Thus, every 
member of the treatment group is matched with one or more people in a comparison group drawn from 
the same recruitment source (adult education, developmental education, or CTE) who did not have the 
opportunity to participate in AO. The procedure creates matches so that the treatment and 
comparison groups are as similar as possible with respect to characteristics observed in the data. The 
outcomes for the matched comparison groups serve as the best possible estimate of what AO 
students would have experienced in the absence of AO.’” For example, the adult education 
comparison group in Kentucky experienced close to zero earnings growth while the adult education 
comparison group in Kansas had particularly strong earnings growth. Those experiences represent the 
likely trajectories of AO students from those respective student populations (given their characteristics 
and the local labor market) if they did not have access to AO. 


Overall, 4,966 students were enrolled in AO who were matched to postsecondary records and thus 
had enrolled in a for-credit course at an AO college in the calendar year 2012 to 2014 implementation 
period in Illinois, Kansas, and Kentucky and the fall 2012 to summer 2015 implementation period in 
Louisiana. That number is smaller than was reported in the implementation research because it 
excludes students in noncredit programs (who were particularly common in Louisiana). Administrative 
data analysis commonly yields smaller numbers of participants than those self-reported by programs 
because of timing, definitional differences, or data deficiencies. Ultimately, 4,760 AO students 
remained in the treatment group after matching to earnings records and other modest sample 
adjustments; these students were then matched to students in a much larger comparison pool based on 
the variables listed in box 1.18 
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BOX 1 
Variables Used for Matching 


The research team used the following extensive set of variables to estimate propensity scores. In 
addition, indicators of missing variables were included where applicable. 


= Local area unemployment rate for the metropolitan statistical area 

= Semester of enrollment in AO (treatment group) or start of most recently observed college 
spell (comparison group) 

= Credential: Indicator variables for if a student earned an HSE credential before enrollment, if a 
student earned a high school diploma before enrollment, and if a student earned greater than 
an HSE credential or high school diploma before enrollment (such as an associate’s or 
bachelor’s degree) 

= Student had prior postsecondary experience 

= Student had prior adult education experience 

= Age and age squared 

= Female 

= Race or ethnicity: white, black, Hispanic, other race, or missing race 

= Single parent 

= Student received a Pell grant 

= Instructional area of enrollment based on Classification of Instructional Program (CIP) code: 
Health care, manufacturing or mechanic trades, other AO occupational area, other non-AO 
occupational area 

= _NRS functioning levels based on adult skills test (adult education students only): Indicator 
variables for scores of 1 or 2, 3,4,5,6 

= Postsecondary test scores (non-adult education students only): Standardized postsecondary 
score (COMPASS, ACT, SAT, ACCUPLACER) and standardized postsecondary score squareda 

= Total number of quarters employed in eight quarters before enrollment 

= Total earnings in eight quarters before enrollment measured in the following four variables: 
fifth to eighth quarters before, third and fourth quarters before, second quarter before, first 
quarter before 

= Employed in two quarters before enrollment: second quarter before and first quarter before 

= Predicted probability of college (adult education only) 


@ Scores are standardized to have a mean of zero and a standard deviation of one. 


After matching students using propensity scores, the research team estimated the effects of AO 
participation on educational and earnings outcomes. Appendix D contains the balancing tests after 
matching, which show that in each state and subpopulation, the treatment and comparison groups are 
fairly well matched by the propensity score analysis alone. Most baseline characteristics are balanced 
after the match, although several remain unbalanced. Because not all baseline conditions are balanced, 
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the impact estimates control for all baseline characteristics in a regression. A regression approach 
adjusts for any remaining observable differences in the treatment and comparison groups and increases 
the precision of the estimates. This approach may not account for unobserved differences between the 
groups, such as intrinsic motivation or strength of workforce-readiness skills. However, matching 
individuals based on each person’s test scores and employment and earnings history likely aligns 
students on many of those difficult-to-measure characteristics. 


Assuming the groups are well-matched on characteristics before AO participation and all baseline 
characteristics are included in the regression, the regression coefficient on the AO participation yields 
the estimated impacts of AO on educational and earnings outcomes. 


Data Sources 


This analysis used administrative data from at least three sources in each state: the state adult 
education program data system, the state’s college data system, and the state unemployment insurance 
earnings records. States coded each individual in the datasets as being an AO participant or not an AO 
participant. Records were collected for students who enrolled in AO programs, college programs, or 
adult education programs in the first three years of the grant period, which began in the spring 2012 in 
Illinois, Kansas, and Kentucky and fall 2012 in Louisiana. AO programming was relatively new during 
this observed period, and the most robust follow-up data are available only for the earliest cohorts. 
Therefore, this is an early look at the impacts of AO. The impacts may change as the initiative matures, 
but that cannot be observed in this analysis. Students were tracked through the latest available date, 
which was often early in calendar year 2016. That provided an opportunity for up to 12 quarters (three 
years) of follow-up for the earliest cohorts and at least 3 quarters of follow-up for the latest cohorts. 
Although those periods of follow-up were sufficient to obtain substantively useful results, a longer 
observation period for more cohorts would strengthen the analysis and confidence in the conclusions. 


One drawback of unemployment insurance data is that they only capture earnings in covered 
employment covered by unemployment insurance within the state. Although the data provide earnings 
records for the vast majority of workers,’ they do not capture earnings for workers who are employed 
across state lines, for some religious and government entities (including the military), for railroads, as 
sole proprietors, or “off the books.” Although relying on unemployment insurance records may produce 
too low of an estimate of employment and earnings, the exclusion of records from some groups is 
unlikely to cause a bias in the impact estimates because the less-than-complete coverage of earnings 
are unlikely to affect AO and non-AO students differently.2° 


The state offices administering the AO grant worked with the Urban Institute and the state labor 
agency to link education and earnings data and then de-identify the records of AO and comparison 
group members so that the research team did not receive students’ personally identifiable 
information.2! Instead, the research team received records with a unique identifier that was not a Social 
Security number or college identification number, and that identifier allowed for linking across files. 
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Analysis Sample 


Table 2 summarizes the steps taken to produce the final analysis sample included in this report. The first 
row of table 2 provides the starting AO sample. This is the number of students in each state’s dataset 
that were flagged as AO students. This number may include students who enrolled before or after the 
three-year analysis period, which ran from the beginning of spring 2012 through the end of fall 2014 in 
Illinois, Kansas, and Kentucky and from the beginning of fall 2012 through the end of summer 2015 in 
Louisiana. 


TABLE 2 
Sample Sizes for AO and Comparison Group Students, by State 


Illinois Kansas Kentucky Louisiana 
AO Comp AO Comp AO Comp AO Comp 
Starting AO sample 1,220 7,474] 2,210 40,375 | 1,372 330,374 | 2,144 226,389 
Sample linked to postsecondary records 952 4,456] 1,733 29,466 | 1,372 229,019 | 556 8,587 
Sample linked to postsecondary and Ulrecords 931 4,131] 1,704 28,889 | 1,362 20,226] 440 3,015 
Sample before matching 867 4,129] 1,699 12,595 | 1,360 18,791} 440 3,015 
Sample after matching 867 4,129] 1,698 12,595 | 1,356 18547] 440 3,015 


Note: AO = Accelerating Opportunity initiative; comp = comparison group; Ul = unemployment insurance. 


The research team only analyzed students with postsecondary education records who enrolled in at 
least one credit-bearing class during the analysis period. As a result, “no-shows,” or students who signed 
up for AO but never enrolled in college during the observed period, are excluded from the analysis, as 
are individuals in the comparison group who did not take a college course.?? Also excluded are students 
who were missing critical identifying information in their adult education records, making it impossible 
for the state office to locate them in the college data even if they did enroll in a course. In Louisiana, 
about 58 percent of the AO sample is not part of the analysis because they were enrolled in noncredit 
postsecondary programs. The exclusion of noncredit programs is appropriate because 1) a core element 
of AO was that AO pathways award “some college-level professional-technical credits, which must be 
recorded in a transcript the quarter or semester in which they are earned” (JFF 2015); and 2) Louisiana 
tracks progress in noncredit programs using a different data system that could not reliably match 
individuals to earnings data because adequate identifying information was not available to the research 
team. 


After linking cases with postsecondary records, the next step was to remove students who were not 
eligible for the analysis. Many such removals were from the comparison group. Students in the 
comparison group in classes with at least four AO students who were within their first four courses 
were removed because it was assumed that these non-AO students were taking classes that were part 
of an AO pathways program.”? The implementation research revealed that non-AO students taking 
classes within AO pathway areas in AO colleges received many of the same services as AO students, 
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including exposure to team teaching and support of an AO navigator, who usually extended their 
services to non-AO students in AO classes. 


Another adjustment to the sample before linking the data files was to remove the small number of 
remaining English-language learner students from AO and the comparison group.”* These students 
have different characteristics from adult education students in their education and labor-market 
trajectories,° and it would be better to match them to an English-language learner comparison group. 
However, the small number of English-language learner students in each state made this analysis 
infeasible. In addition, test scores for English-language learner students were not directly comparable 
to the test scores used to measure adult basic skills, further complicating a pooled analysis. 


The next step was to link students with unemployment insurance administrative data, which has 
quarterly earnings records for nearly all workers in a state.”° The only reason that a student would not 
match to unemployment insurance records (other than not having any earnings) is if they worked for an 
employer not covered by unemployment insurance (as discussed above) or did not have a valid Social 
Security number.”’ If a student did have a valid Social Security Number but did not match to earnings 
records, the research team assumed that he or she did not have any employment history in a job 
covered by unemployment insurance during the relevant period. Because unemployment insurance 
records are collected at the state level, it is also possible that students who do not match to the earnings 
records in one or all of the study quarters were employed in another state, as noted previously. 


After developing the final linked dataset, the research team matched the AO students to 
comparison group cases. Some comparison group cases did not match to any treatment cases and 
dropped out of the analysis. In Kentucky, four AO students dropped out of the analysis because there 
were not comparable non-AO cases, but such a change to the dataset was too small to have any effect 
on the estimates produced. In the end, 4,361 AO students were retained in the analysis across the four 
states. 


Adjustment for Comparison Group’s College Enrollment 


This analysis compares adult education-recruited AO students to adult education students who 
enrolled in at least one college course. The research team chose this comparison group for three 
reasons: 1) AO students are motivated and low-barrier enough to express an interest in college and 
enroll in AO, 2) much of the necessary data for the analysis are in the college files, and 3) it followed the 
precedent of the |-BEST research. 


If all AO students would likely have enrolled in college if AO did not exist, then comparing AO 
students to adult education students who took a college course of their own accord would be sufficient 
because both groups had an interest in and motivation to enroll in college CTE courses. However, AO 
may also promote access to college among adult education students who otherwise would not have 
enrolled because of academic, motivational, or financial barriers, which the program may have 
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alleviated. In the AO implementation research, several students who were interviewed stated that they 
did not think college was an option for them until they heard about AO, which indicates that AO played 
arole in college access, at least for some students. If many AO students who came from adult education 
would not have enrolled in college if not for AO, then comparing them to adult education students who 
were able to access college without the assistance of a program such as AO (and therefore were both 
particularly motivated and had relatively low barriers to college access?°) may lead to underestimates of 


the impacts of AO. AO students are probably realistically in the middle between a “typical” non-college- 
going adult education student and a very motivated, low-barrier adult education student who enrolls in 


college on their own. 


The research team attempted to address this issue by estimating a “predicted probability of college” 
measure for the adult education population and included this measure in the propensity score matching 
analysis. The team estimated this probability by implementing a logistic regression analysis for the 
entire adult education population (except for AO students), both those who enrolled in college and 
those who did not, to see which characteristics within that population predict college enrollment. This 
analysis excluded AO students because AO is an extra support that may confound the results. The 
research team used a scaled down model to predict college enrollment based on the availability of the 
variables in the adult education datasets. The variables used were adult education enrollment date, 
receipt of a high school credential, age, gender, race or ethnicity, and single-parent status. 


After modeling college enrollment for the entire adult education population, the research team 
used the model to predict college enrollment for the AO and comparison groups. This predicted 
probability of college enrollment was then included in the matching model for the adult education 
population so that AO participants were matched with members of the comparison group who had a 
similar likelihood of going to college. 


Though the measure of predicted probability of attending college may not address all conditions 
that cause an adult education student to enroll in college, it does help alleviate some of the issue that 
members of the AO treatment group and the adult education comparison group have different 
propensities to attend college. Addition of the probability of attending college as a matching variable 
does not substantively change the results from earlier analyses that did not use this variable. 


Separation by Recruitment Source 


Though the analysis accounts for many measured characteristics, it is also important to try to align the 
unmeasured differences between students. One way to do this is to account for AO participants’ 
recruitment source. Some students came to AO through the adult education system; others enrolled in 
AO while participating in developmental education or while enrolled in CTE (but with low initial basic 
skills test scores). Adult education students tend not to have high school diplomas or HSE credentials 
and face particular challenges accessing postsecondary education. Students with low basic skills in CTE 
or developmental education programs tend to have secondary school credentials and have fewer 
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barriers accessing college, but they may struggle to succeed in college-level courses. Students recruited 
through different sources are likely to differ in unmeasured student characteristics. Therefore, the 
analysis matches AO students who were recruited from adult education with non-AO adult education 
students; those recruited from developmental education and from CTE are matched with similar 
students from those sources, respectively.?? The impact results are reported for AO students (matched 
to the appropriate comparison group) by state and recruitment source. The impacts by recruitment 
source capture different effects of the AO model for different populations. 


Outcomes Measured 


This report examines students’ education and labor-market outcomes. The education outcomes are 
= — the number of credentials earned through the college,°° 
= whether the student earned any credential earned through the college (yes/no), 
= the number of credits earned,*? and 
= whether the student earned more than 12 credits (yes/no). 
The labor-market outcomes are 


= the student’s employment in each of the 12 quarters following enrollment (or for as long as data 
are available), and 


= the student’s earnings in each of the 12 quarters following enrollment (or for as long as data are 
available). 


The labor-market outcomes are predicted for each quarter following enrollment. Labor-market 
data on all students were available for 3 to 12 quarters of postenrollment earnings. Education data 
were available for 2 to 11 semesters following enrollment. Time censoring limited the data observed for 
later cohorts. As a result, the sample size decreases beyond the third or fourth quarter, and the longer- 
term outcomes reflect the experiences of the earlier AO cohorts.°? The section “Cohort Effects” 
provides results for each cohort at colleges that will be part of the cost-benefit analysis. 


The outcomes have a few important caveats. First, the research team recognizes that credits, 
credentials, and labor-market outcomes do not represent the full range of potential benefits of a 
program such as AO. A larger goal of AO (and many similar programs) is to help students live better, 
more fulfilling lives. Although education and labor-market outcomes contribute to quality of life, the 
analysis does not account for potential benefits such as self-esteem, family stability, health, or other 
aspects of personal well-being. Further, many states and colleges used AO as an opportunity to 
implement systemic changes to improve opportunities for adult education and other low-skill students. 
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Elements of systems change are not captured in this analysis, though some aspects are described in the 
implementation findings (Anderson et al. 2016). 


Second, the credentials counted are college credentials that may or may not have value in the labor 
market. Though some states, such as Illinois, attempted to track third-party-awarded credentials (such 
as industry licenses), tracking was not consistent enough across states to include such credentials in the 
analysis. The cross-state variation in credentials is another reason the research team did not pool 
results across states. College-awarded credentials vary widely; a six-week certified nurse aide (CNA) 
certificate is a credential as is a two-year associate’s degree. The research team did not attempt to 
categorize credentials but instead presumed that more valuable credentials would be reflected in 
students’ labor-market outcomes. Therefore, the “counting” of credentials should be interpreted with 
appropriate caution. 


Finally, the research team determined it was not possible to measure skill gains, though this was a 
measure of interest in the previous I-BEST research. Many college staff members noted that it was 
often difficult to convince AO students to return to adult education to take basic skill exams once the 
students began in college classes. That is reflected in the data, where posttest scores are lacking for 
many AO students. Because differences between students who did retest and those who did not could 
introduce bias, the research team could not reliably measure the impact of AO on changes in basic skill 
scores. 
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Description of AO Students 


AO served adults from diverse backgrounds, education levels, and recruitment sources. 
This chapter opens with a description of AO recruitment across states. It then describes 
AO student characteristics across the initiative based on administrative data and the 
recruitment practices in each state.?3 (Appendix B contains detailed characteristics of 
AO students in each state.) It closes with an examination of the achievements of 
students by occupational area to provide additional insight into AO implementation and 


helps contextualize the impact results. 


State Recruitment Practices 


Illinois and Louisiana recruited entirely from adult education programs or similar populations, as 
described by Anderson and colleagues (2016). That strategy was closer to the original intent of AO to 
focus on adult education students, and the states worked diligently to recruit this often hard-to-serve 
population. Therefore, in Illinois and Louisiana, AO students were matched only to similar adult 
education students, and the AO impact results in those states are reported only for the adult education 
population. 


Kansas and Kentucky, however, took different approaches to AO recruitment in that each had two 
primary recruitment sources. The impact analysis matched AO students from each source with 
comparison group members from the same recruitment source to correct for inherent differences 
between adult education and other college students that are not reflected in their measurable personal 
characteristics. The following descriptions of recruitment practices in Kansas and Kentucky provide a 
deeper explanation of the context and recruitment strategies. 


Kansas AO Recruitment 


Kansas recruited AO students from both adult education (27 percent of the treatment group) and 
college sources, particularly CTE classes (73 percent of the treatment group).°4 The state and colleges 
were motivated to draw on the college population in addition to the adult education population for two 
reasons: 1) staff saw low-skilled AO-eligible college students as a group that could benefit from the 
extra services AO would provide; and 2) staff were motivated to reach the target of 3,600 credentials 
but challenged by the termination of the ability-to-benefit provision in Pell grants, which had allowed 
those without high school diplomas or HSEs to receive federal financial aid to support tuition costs. 
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After the AO grant, the state refocused recruitment into AO on the adult education population, 
supported by new funding from the state legislature (Anderson et al. 2016). 


Recruitment from these two primary sources varied. Although staff guided adult education 
students without high school credentials into AO based on their own assessment of the student’s 
workforce readiness skills and motivation, staff enrolled many CTE students into AO by going into 
already-formed relevant CTE classrooms and testing all students to determine AO eligibility. According 
to staff at several Kansas colleges, most CTE students had basic skills deficiencies and therefore met 
AO eligibility requirements, even though most had high school credentials. In those cases, AO staff gave 
AO resources (including a team teacher) to the whole classroom, including those who tested above AO 
eligibility.>° 


This report presents separate analyses for each recruitment source in Kansas because the 
characteristics that predicted AO participation and the nature of the AO intervention differed 
substantially for students from each recruitment source. For the CTE group, the AO intervention 
generally consisted of the introduction of team teaching into their classes and the extra support offered 
by acollege or career navigator. Conversely, for adult education students, AO staff may have played a 
role in addressing obstacles to college enrollment and success and staff included more intensive 
coaching and barrier remediation. Discussions with staff and administrators during the implementation 
research supported the notion that adult education students usually have higher barriers to college 
entry than the CTE group. The section “Adjustment for Comparison Group’s College EnrollmentError! R 
eference source not found.” describes one adjustment for possible differences in college access among 
the adult education treatment and comparison groups by modeling the likelihood of attending college 
for the entire adult education population in the state and including the predicted probability of college 
as avariable for matching between the adult education treatment and comparison groups. 


Kansas did not provide an explicit flag identifying AO students who were recruited from CTE 
classes. Instead, this population was approximated using a strategy developed in consultation with the 
state. Staff at the colleges were required to enroll all Kansas AO students in the adult education data 
system. If the date of college enrollment for an AO student preceded the date they were enrolled in the 
adult education data system, the research team assumed that that student was recruited into AO froma 
CTE class or another college source (shortened to “CTE” because that characterizes the majority of 
non-adult education students). If, however, the AO student was enrolled in college after their 
enrollment into the adult education data system, then the research team assumed that student was 
recruited from adult education and only subsequently enrolled in college after starting AO. 


Kentucky AO Recruitment 


Kentucky colleges recruited from both adult education and developmental-education programs at 
community colleges. As in Kansas, colleges tended to recruit students with a high school credential for 
AO primarily because of difficulties finding financial assistance for students without high school 
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credentials after the termination of the Pell grant’s ability-to-benefit provision in July 2012. Those 
funding challenges are discussed by Anderson and colleagues (2016). Further, state administrators 
indicated that the change in the GED test in January 2014 made it more challenging to recruit adult 
education students because they had to put more concentrated energy toward completing their HSE 
and were more hesitant to coenroll in college. The state office reports that it has increased its focus on 
the adult education population following the end of the evaluation period. 


This analysis deals separately with students recruited from adult education and those recruited 
from developmental education. As in Kansas, the observed characteristics that predicted AO 
participation differed substantially between the two groups. Moreover, because all students in 
developmental education had already enrolled in college or were at the point of college enrollment 
when they were recruited into AO, the research team expects that AO would have little or no impact on 
the ability of the treatment or comparison group to access college. For AO students recruited from 
developmental education, the AO intervention generally consisted of allowing students to enroll in for- 
credit courses instead of primarily noncredit developmental courses, supported by team teaching and a 
college navigator. For adult education students, AO staff may have played a larger role in addressing 
obstacles to college enrollment and success, and the treatment and comparison groups may have 
differed in motivation. See the discussion below of the “predicted probability of college” measure, which 
the research team developed to partially address potential differences in college access for the adult 
education treatment and comparison groups. 


The dataset provided by Kentucky did not contain an identifier of student recruitment source. 
Consultations with state officials suggested that all or nearly all students recruited from developmental 
education had a high school credential at AO entry. Moreover, the officials suggested that, given the 
pressure on adult education students to complete their HSEs before enrolling in AO in order to qualify 
for financial aid, students without an HSE at entry or who earned an HSE in the semester immediately 
before AO enrollment were most likely recruited from adult education programs. Because of the 
absence of data on recruitment source in Kentucky, groups are separated by education status at AO 
entry.*¢ According to state officials, the education distinction largely captures the difference in 
recruitment source. Using this approach, about 9 percent of participants were recruited from adult 
education, and the remaining 91 percent were recruited from developmental education. The analysis 
results are reported for each of these populations and for the weighted average of all AO students. 


Selected Characteristics 


Table 3 presents descriptive information about AO students overall and in key occupational areas. To 
determine the occupational areas in which students were enrolled, the research team used course and 
credential data to classify AO students into three broad categories: health occupations (53 percent), 
manufacturing occupations (36 percent), and other or unidentifiable occupations (12 percent).°” Figure 
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3 displays some differences in key characteristics among states. Appendix B includes full student 
characteristics for each state. 


TABLE 3 
Selected Variables for AO Students, by Occupational Pathway 


AO AO students 
students AOstudentsin in other or 
AO inhealth manufacturing unidentifiable 

students pathways pathways pathways 

Female (%) 57.7 88.1 12.2 58.2 

Average age at intake 28.3 28.2 28.4 28.0 

White (%) 57.9 59.8 59.2 45.1 

Black or African American (%) 22.0 19.4 21.0 35.1 

Hispanic or Latino (%) 11.0 10.2 11.9 12.5 

High school diploma or HSE at entry (%) 62.2 64.0 67.9 37.8 

Greater than high school at entry (%) 8.6 12.0 5.4 1.6 

Pell grant recipient (%) 35.6 33.2 42.4 26.3 

Predicted probability of attending college (%)? 26.3 27.7 23.3 28.3 

Average quarters employed in prior two years 3.6 3.9 3.4 2.6 
Sample size 4,361 2,293 1,555 513 


Note: HSE = high-school equivalency. 
® Adult education students only. 
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FIGURE 3 


A Comparison of Selected AO Student Characteristics across States 
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As shown in table 3, over half (58 percent) of AO students across all states were women. In each 
state, the share of female students was much higher in health pathways (88 percent) and much lower in 
manufacturing pathways (12 percent). Fifty-eight percent of AO students were white, 22 percent were 
black or African American, 11 percent were Hispanic or Latino, and the remainder were identified as 
“other race/ethnicity.” Across the states, 63 percent of AO students had a high school credential and 9 
percent had a prior postsecondary credential, though the percentage in each state and from different 
recruitment sources varied widely. Over one-third (36 percent) of AO students received Pell grants. It is 
surprising that Pell grant receipt is not higher, given that so many students possessed high school 
credentials at AO entry. However, state administrators noted that some programs (such as short 
welding programs) were not eligible for Pell grants, even for students with high school credentials. 
Further, some colleges offered scholarships to students so that they did not need to apply for financial 
aid during the AO pathway period. The average AO student worked for 3.6 of the 8 quarters before 
enrollment, with stronger employment histories for students in health pathways than in other 
occupational areas. 


Important variations in the AO population are apparent in figure 3. The Kansas adult education 
population had the highest share of female students (77 percent) of any state or recruitment source. 
Louisiana also had a relatively large share of female students (64 percent). Kansas CTE and Kentucky 
developmental education students were less likely to be female than other states or recruitment 
sources, at 53 and 55 percent respectively. This is likely related to the distribution of enrollment among 
occupational areas. 


The average age of AO students was fairly comparable across states, though AO students in 
Louisiana, with an average age of 26, were somewhat younger than students in other states. Kentucky 
students from both adult education and developmental education, at nearly 30 years old, were 
somewhat older. 


The racial composition varied widely, largely reflecting variations in the racial and ethnic 
populations of the states. Although 84 percent of developmental education AO students in Kentucky 
were white, only 37 percent of AO students in Louisiana were white. Many more AO students in Illinois 
and Louisiana were black or African American than in other states. Illinois and Kansas had relatively 
larger shares of students who were Hispanic or Latino. 


As would be expected, students recruited from adult education in each state were less likely to have 
a high school credential than those recruited from developmental education in Kentucky or CTE in 
Kansas. In Kentucky, all students recruited from developmental education had a high school credential 
by definition. Notably, a relatively large proportion of adult education students in Kansas had a high 
school credential, but that was mainly a result of students earning their credentials shortly before AO 
enrollment. In Kansas, a relatively large proportion of CTE students had a postsecondary credential 
before AO enrollment. That statistic is expected given that Kansas primarily recruited students from 
already-formed college classes. 
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The predicted probability of attending college applies only to students recruited from adult 
education. Students in Illinois had the highest predicted probability of attending college; students from 
Louisiana had the lowest. However, the predicted probabilities are difficult to compare across states 
because they are principally determined by the prevalence of college attendance among adult 
education students generally in a state. As a hypothetical example, an AO student with a low predicted 
probability of attending college in Louisiana may have a much higher predicted probability of attending 
college if they had the same characteristics but lived in Illinois. 


Finally, the average number of quarters worked in the prior two years varies little across states. 
Students from Kansas have relatively more robust recent work histories than students from adult 
education. Students in Illinois had the least recent work history of any of the states or recruitment 
sources. Note that the differences are smaller in appearance in the graphic because of the scale; the 
maximum number of quarters a student could have been employed in the prior two years is 8, while the 
scale continues to 100. 


Changes in Characteristics over Time 


Figure 4 show the changes in the characteristics of AO students across the states by program year. In 
Illinois, Kansas, and Kentucky, a program year corresponded to calendar years; in Louisiana a program 
year corresponded to an academic year because of a delay in starting the AO grant. Over the course of 
implementation, AO students were significantly more likely to be men, despite sustained majorities of 
women. Otherwise, the characteristics of students across the states displayed variability over time but 
no notable trends. 
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FIGURE 4 
Total Enrollment by Academic Year, All States 
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Notes: Percentages are computed for students for whom data are available; missing values are excluded. Years correspond to 
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technical education; HSE = high school equivalency; Dev. ed. = developmental education. 
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Basic-Skill-Level Variation 


Basic skill levels are of interest because such levels were the primary method of determining student 
AO eligibility. The AO guidelines specified that the students should score on basic skill tests in the NRS 
functioning levels 4 through 6 (and English language NRS functioning levels 5 and 6, though those few 
students are excluded from the analysis). Students with those scores have high intermediate basic 
education to high adult secondary education skill levels. 


The implementation research revealed that colleges applied cutoff criteria for AO in different ways. 
JFF and the state system offices did not require that colleges modify or amend existing assessment 
requirements for students. Some colleges admitted students into AO based on their composite scores; 
others admitted students to AO if one of their subscores in reading, writing, or math fell within the 
eligible range, even if the student’s composite score was outside the eligible range. Some programs 
determined eligibility based on the subscore that staff determined was most relevant to the program 
area (such as math for manufacturing). Such variations in eligibility policies led to a wide range of basic 
skill levels among AO students. Skill level differences across states may lead to differences in AO 
impacts if AO affected relatively high-skill students differently than low-skill students. Similarly, 
differences in other student characteristics may lead to differences in impact if the AO intervention is 
more effective for certain types of students than others. 


Figure 5 describes the composite NRS functioning levels for AO students by state and recruitment 
source.?8 The NRS functioning levels are based on the most recent scores before AO enrollment. 
Though basic skills test scores would typically only be available for participants in adult education, most 
AO students were assessed for their NRS functioning level regardless of recruitment source. Most 
scores were reported in the state’s adult education data system, but some populations of AO students 
have high missing rates on NRS functioning levels. It is possible that these students were assessed but 
their scores were not uniformly recorded in the adult education data systems, preventing states from 
providing the scores to the research team. Alternatively, the colleges may have based eligibility 
decisions on a different test, such as an entrance examination used in many colleges (e.g., ACT or 
COMPASS). Such exams do not produce NRS functioning levels and often measure different 
competencies than adult basic skills exams. 


Among the AO participants with NRS test data, the vast majority scored at NRS functioning levels 
4-6, the levels targeted by the designers of AO. The most frequent NRS functioning level was 5, or a 9th 
to 10th grade functioning level. About 12 percent of participants tested below NRS functioning level 4. 
Illinois and Kentucky adult education had a larger share of students who tested below NRS functioning 
level 4 than the other states, at about 20 percent and 29 percent, respectively.’ Although over 60 
percent of AO participants had completed a high school credential (Kansas and Kentucky participants 
recruited from developmental education or CTE classes, in particular, were very likely to have a high 
school credential), a small number tested at the NRS functioning level 6 or above, equivalent at least an 
11th grade functioning level. One state noted that the discrepancies between high school completion 
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and test scores may be partially explained by a substantial time gap between a student’s high school 
experience and their basic skills testing for AO. 


FIGURE 5 
Basic Skill Levels for AO Students at Enrollment, by State and Recruitment Source 
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Note: NRS = National Reporting System educational functioning level. Data are for nonmissing values. The missing rates were as 
follows: IL: 17 percent, KS adult education: O percent, KS CTE: O percent, KY adult education: 18 percent, KY developmental 
education: 2 percent, LA: 33 percent. 


Outcomes Achieved 


Table 4 displays the credit and credential achievements of the full AO sample and the sample used in 
this evaluation. Data were available for students 2 to 11 semesters following AO enrollment, depending 
on cohort. Students in the AO impact sample enrolled in at least one college course, were not English- 
language learners, had a valid Social Security number, and were retained in the sample after propensity 
score matching. Table 4 presents descriptive statistics only and does not represent net impacts of AO; 
net impacts are presented in the next chapter. Table 4 is based on the administrative data provided by 
the states. The data in table 4 may therefore differ from the AO sample reported in the grantee surveys 
and in other grant reporting. 
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TABLE 4 
AO Enrollment, Credits, and Credentials by Sample and State 


Full AO Sample AO Impact Sample 

Cred. per Crdn. per Cred. per Crdn. per 

Enrollees Cred. Crdn. _ Enrollee enrollee |Enrollees Cred. Crdn. enrollee enrollee 
IL 952 10,094.5 423 10.6 0.4 867 9,300 385 10.7 0.4 
KS 1,733 38,401 3,636 22.2 2.1 1,698 37,647.9 3,567 22.2 2.1 
KY 1,372 25,596 2,247 18.7 1.6 1,356 25,499 2,242 18.8 1.7 
LA 515 5,010.5 482 9.7 0.9 440 4,246.5 378 9.7 0.9 
Total 4,572 79,102 6,788 17.3 1.5 4,359  76,693.4 6,572 17.6 1.5 


Note: AO = Accelerating Opportunity initiative; cred. = credits; crdn. = credentials. Samples are students enrolled in the first three 
years of AO implementation. 


Inall, the states enrolled 4,572 students in AO, awarding 6,788 credentials and 79,102 credits 
through the end of 2014 in Illinois, Kansas, and Kentucky and summer 2015 in Louisiana. Kansas had 
the highest enrollment, at 1,733 students; Louisiana had the lowest enrollment at 515 students (mainly 
because about three-quarters of the 2,144 students flagged as AO in the Louisiana data did not enroll in 
credit courses, leading to their exclusion from the sample). Kansas also had the highest number of 
credits per enrollee and the highest number of credentials per enrollee. 


Outcomes by Occupational Area 


Because students may change their mind about their occupational area after enrolling in AO based on 
their experience in the program, the research team did not generate separate impact estimates by 
occupational area.*° However, an examination of descriptive outcomes by occupational area is useful. 
Figure 6, figure 7, figure 8, and figure 9 describe the education and labor-market outcomes for AO 
students by occupational area. Unlike the impact estimates in the next chapter, which include people 
with no earnings, the earnings presented in figure 9 are only for students who were employed. Figure 9 
is limited to students with earnings to represent the approximate amount of an employed person’s 
quarterly “paycheck,” while the impact estimates capture average earnings gains, both from entry into 
the labor market and for those already employed. The estimates in the following figures are only for the 
AO group and do not represent impacts of AO. 


Occupational areas were determined as described earlier. AO students varied considerably across 
the occupational pathways. In some cases, variation in student characteristics is greater between two 
occupational pathways within a state than it is between two states. 


In most of the states, the average number of earned credentials varied little across occupational 
areas.*1 However, state differences did emerge. For example, students in Kansas earned a larger 
number of credentials on average across all occupational areas than did students in the other three 
states. In states except Illinois, students in manufacturing earned more credentials on average than 
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students in other occupational areas; in Illinois, students in health and manufacturing earned 
approximately the same number of credentials on average. Students in “other” pathways in Illinois and 
in health pathways in Louisiana earned a particularly small number of credentials on average. The 
differences in credential attainment by occupational area have been caused by differences in the 
structure of pathways, differences in the persistence of students to earn additional credentials within 
the observed period, or a combination of both. Further, if many health credentialing programs are 
tracked in the noncredit system in Louisiana, that may explain the low number of average health 
credentials in that state because those data come only from the for-credit data system. Examples of 
credentials included college-awarded certificates for CNAs, CNC operators, welding technicians, and 
automotive technicians. 


Credit attainment varied widely among states and occupational areas, but the patterns across 
occupational areas are not consistent among the states. In Kansas, students in health occupations 
earned on average just over half the number of credits as students in “other” occupational areas earned. 
In Kentucky, however, students in health pathways earned slightly more credits on average than 
students in manufacturing, who earned slightly more credits on average than students in other pathway 
areas. In Illinois and Louisiana, students in manufacturing earned the most credits, followed by health. 
Although Kansas students exhibited notably higher credit attainment than students in other states, the 
difference represents only about one more semester of courses. High credit attainment in Kansas 
across occupations may reflect the higher level of initial skills of its AO students, as indicated by school 
attainment and NRS functioning levels, or it may reflect differences in the structure of credits within the 
state. For example, the Kansas Board of Regents undertook a concerted effort to align all of their credit- 
bearing CTE programs across colleges and with the needs of employers beginning in 2010. As a result, 
Kansas offers more programs for-credit than many other states, but these credits are well aligned with 
labor market needs because of the state’s efforts. This may affect credit accumulation as well as labor 
market outcomes in the state. 


Turning to employment, students in Kentucky had the lowest employment rates in all three 
occupational areas, Kansas had the highest employment rate in health and manufacturing, and 
Louisiana had the highest rate in other occupational areas. Similarly, Kansas had the highest level of 
earnings in health and manufacturing, while students in Louisiana had the highest earnings in “other” 
occupational areas, including information technology, early childhood education, business, and 
automotive pathways. Illinois had the lowest level of earnings in health and other professions, while 
Louisiana had the lowest earnings for manufacturing. Note that those data only represent earnings for 
those who were employed. 


Although health professions generally had the highest rates of employment, students in 
manufacturing professions generally had the highest earnings (except in Louisiana). That may reflect 
the sector’s characteristics and the workers’ characteristics, because students in manufacturing 
pathways tend to be overwhelmingly men (12 percent across all states) while students in health 
pathways tend to be overwhelmingly women (88 percent across all states). The manufacturing 
profession’s high earnings may also reflect the levels of positions targeted by AO pathways. Health 
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pathways generally targeted more entry-level, low-paid positions such as CNA, phlebotomists, and 
home health aides. Manufacturing pathways may have targeted mid-level, well-paid occupations, such 
as welders and computer-aided manufacturing technicians. Anderson and colleagues (2016) describe 
the pathways in more detail. The earnings data reported here may reflect patterns of interstate 
commuting. If people in certain occupations or state-occupation combinations are more likely to work 
across state lines, their employment and earnings would not be reflected in these data. 


FIGURE 6 
Number of Credentials Earned Per Student by AO Students, by State and Occupational Area 
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Number of Credits Earned per Student by AO Students, by State and Occupational Area 
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FIGURE 8 
Percentage of AO Students Employed, Quarterly by State and Occupational Area 


Quarter O Quarter 1 ™ Quarter 2 @ Quarter 3 @ Quarter 4 @ Quarter 5 @ Quarter 6 
(entry) 
100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 
0% 
2 fol) . on oD 7 € fol) 7 x for) = 
= 8 o = g O x £ o = g ° 
oO oO oO oO 
5 5 5 Ss 
Cc Cc Cc Cc 
oO oO oO oO 
= = = = 
IL KS KY LA 
FIGURE 9 


Quarterly Earnings for AO Students of Those with Any Earnings, Quarterly by State and 
Occupational Area 
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Note: Some values in Kentucky were redacted due to cell size restrictions. 
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Summary of AO Student Description 


Information from the implementation study on state recruitment practices has been important in the 
design of the impact study because it revealed that two of the four states have drawn substantial 
portions of their AO population from non-adult education sources. 


Overall, AO students are somewhat more likely to be women and older than traditional college 
students, and they more or less reflect the racial and ethnic demographics of each state. Test scores of 
adult education and CTE students in Kansas did not differ much, but both were higher than AO 
participants’ test scores in other states. Differences in test scores for adult education and 
developmental education students in Kentucky were larger, with adult education students scoring 
substantially lower. Louisiana and Illinois AO students, who were all from adult education, had test 
scores similar to those of developmental education students in Kentucky. Student demographic 
characteristics did not change notably over time but did shift to include more men, probably reflecting 
changing pathway offerings at the colleges. Overall, more than 4,500 AO enrollees earned over 79,000 
credits and nearly 6,800 credentials by spring 2015 in Illinois, Kansas, and Kentucky and by spring 2016 
in Louisiana. Students in most states experienced growth in employment and earnings over time, 
though Kentucky had the lowest levels and growth of employment and Illinois had below-average 
earnings in health and other professions. Louisiana had the lowest earnings in manufacturing. In all 
states except Louisiana, manufacturing professions had the highest earnings of the three occupational 
categories examined. 


Kansas appeared to have the most advantaged students as measured by educational attainment, 
prior employment, and NRS functioning level. Students in Kansas earned the largest number of credits 
and credentials and had the highest employment levels and earnings. 


Additional state-specific descriptive statistics are available in appendix B. The variation in 
participant characteristics across states, recruitment sources, and occupational areas requires careful 
adjustments of the comparison group to ensure equivalence on baseline characteristics to produce an 
impact estimate. This variation also shows the importance of identifying impact estimates for each state 
and recruitment source separately. Impact estimates are presented in the next chapter. 
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Impact Estimates 


This chapter presents state-specific results for the four states implementing AO. It 
includes impacts of AO on education outcomes and on employment and earnings. The 
AO model intended to increase the number of credits and credentials that students 
earned and to improve their employment and earnings outcomes. Appendix C contains 


tables with detailed earnings and employment impacts by state. 


All students who are in the impact sample are represented in this analysis. The follow-up period is 
noted for each state, but it allows for 2 to 10 semesters of college enrollment and 3 to 12 quarters of 
follow-up depending on the student cohort and the state data. The Methodology chapter provides a 
detailed description of the analysis approach. 


Expected and Summary Results 


Before presenting the state-by-state summaries, this discussion previews overall patterns and 
summarizes the findings. AO generally had positive effects on the attainment of CTE credentials 
awarded by colleges but mixed effects on college persistence. It also had positive effects on 
postprogram employment and some limited positive effects on postprogram earnings that persisted 
over time for CTE students in Kansas and adult education students in Kentucky but faded in most other 
instances. The effects varied across states and student populations, suggesting that AO exerted 
different impacts on outcomes depending on the population served, local economic context, and 
implementation conditions. 


One would expect that AO’s net impacts on credentials earned would be positive given the 
program’s focus on helping students obtain marketable occupational credentials. AO’s emphasis on 
accelerated learning for those with low basic skills could help students obtain those credentials through 
more purposeful enrollment in courses and credits. The impact results in several states demonstrate 
that the number of credits earned by AO participants is lower than the comparison group even though 
the number of credentials is higher. That finding may be evidence of acceleration because AO students 
earned more credentials even though they took fewer courses and credits. 


Persistence beyond 12 credits would suggest that students continued with college beyond the 
initial pathway. Some students may use their new credentials to become employed during or 
immediately following the pathway period, leaving school for some period of time, and then returning to 
school to seek additional education or career advancement. If such a pattern is common, persistence 
may not be observable within the relatively short follow-up period observed in this study. Other 
research has shown that low-income students (which many AO students are, even though AO eligibility 
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does not explicitly account for income) tend to take a break from college to work or tend to life needs, 
such as child care. Many students, however, do return eventually to college once they have started, 
even years later.4#Overall, AO had mixed effects on persistence in the short term. 


Many of the AO initial pathways lasted one to three quarters. With students still participating in AO 
classes and other services in the initial quarters, one would expect enrollment in AO to lead to short- 
term reductions in work effort. During this in-program period, participants have “forgone earnings” as 
they shift time from work to school. After this initial period of negative impacts, however, one would 
expect AO to increase employment rates and average earnings. Because earnings outcomes take more 
than a year or two to manifest, this report may not capture long-term gains that would emerge witha 
longer follow-up period. On the other hand, programs such as AO often experience fade-out effects, 
where the initial gains fade out as the earnings of the treatment group converge toward the earnings of 
the comparison group.*? 


Later cohorts of students are observed for fewer quarters after enrollment because of the timing of 
the evaluation. As a result, the composition of the sample shifts as the time from enrollment lengthens, 
possibly introducing cohort effects. Impacts might vary by cohort because of differences in their 
characteristics or in program experiences. Program experiences were likely different across cohorts 
because AO efforts were least developed for the earliest cohorts. Further, the characteristics of 
participants and comparison group members may differ across cohorts because those who entered 
college in early 2012 during the economic recovery may have weaker earnings trajectories than those 
who entered college in later years. Therefore, the longer-term effects of AO are only observed for 
students who experienced early program implementation; their experiences and the impacts of AO may 
differ from those of later cohorts. The section “Cohort Effects” below presents impact estimates by 
cohort for the students enrolled at colleges that are included in the cost-benefit analysis. 


Tracking the Comparison Group 


In any impact analysis, the characteristics and experiences of the comparison group matter a great deal. 
The experiences of a well-matched comparison group, as is used in this analysis, represents the best 
estimate of what AO students would have experienced in the absence of AO. If the comparison group 
experienced strong earnings growth, AO students would have likely experienced similar growth even if 
they were not in AO. Conversely, a comparison group with weak outcomes suggests that AO students 
would have had similarly weak outcomes without the intervention. 


If there were alternative, similar programs or services available in the state that AO students would 
have received in place of AO, then the comparison group would have benefitted from these programs 
and there would be very small or no additional impacts associated with AO on top of the already 
available programs. The four states had similar initiatives, such as federal grant programs like the Trade 
Adjustment Assistance Community College and Career Training Grants (TAACCCT) and the Health 
Profession Opportunity Grants (HPOG), at the same time AO was being implemented. Moreover, the 
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state office in Louisiana implemented I-BEST-like career pathways with navigation support across all 
state adult education programming concurrent with AO implementation through their Train to Attain 
initiative. Though Louisiana’s statewide supportive pathways were motivated by AO and related grant 
efforts, their broad availability during the AO evaluation period likely affected the outcomes of the 
comparison group, leading to AO’s minimal and even negative impacts in the state. 


Illinois Impact Estimates 


Illinois Education Impacts 


AO impact estimates on education and earnings in Illinois cover the cohorts of AO participants who 
enrolled from the beginning of the spring 2012 semester through the end of the fall 2014 semester. The 
data on educational activities for this report cover the period through the end of the spring 2015 
semester, allowing for 2 to 10 semesters of follow-up. The Illinois AO impact results on education 
outcomes are presented in table 5. AO generally improved educational outcomes, particularly the 
number of college-awarded credentials earned and the share of participants earning at least 12 hours of 
credit. Specifics are as follows: 


= AOincreased the number of credentials earned for all participants by 0.1 credentials on 
average, a 25 percent increase over the number of credentials earned by the comparison group. 


= AOincreased the probability that AO students earned any credential by 11.0 percentage 
points, an increase of more than 35 percent over the probability of earning a credential in the 
comparison group. 


= AOreduced the average number of credits earned by 0.9 credits. The reduction in the number 
of credits earned might reflect acceleration, as described earlier. 


= AOincreased the probability that participants earned more than 12 credits by 5.5 percentage 
points, a 19 percent increase over the comparison group. 
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TABLE 5 


Education Impact Results for Illinois 


Comparison AO gain over 
group mean AO group comparison 
Outcome outcome mean outcome AO impact (%) 
Number of credentials earned 0.3 0.4 0.1*** 25% 
Any credential earned 31.2% 40.6% 11.0 pp*** 35% 
Number of credits earned 11.2 10.7 -0.9*** -8% 
Earned more than 12 credits 28.7% 36.9% 5.5 pp*** 19% 


Note: N = 4,996; pp = percentage points. Impacts may not equal the exact difference between the comparison and treatment means 
because of rounding and regression adjustment of the impact estimates. 
** 9) < 0.01, ** p< 0.05, *p<0.10 


Illinois Employment and Earnings Impacts 


The Illinois AO impact results on employment and earnings cover AO entrants in spring 2012 through 
fall 2014 and follow them for 3 through 12 quarters (three years). The total sample sizes associated 
with each quarter after enrollment are reported in appendix C. Readers should note that all dollar 
amount averages include students who earned $0 per quarter. The purpose of including cases that had 
zero earnings is to ensure that individuals who move from zero earnings to positive earnings as a result 
of AO have these benefits counted in the impact estimates. 


Overall, AO exerted positive impacts on employment for a substantial period after enrollment, but 
the gains in quarterly earnings are modest and diminish over time. Figure 10, figure 11, figure 12, and 
figure 13 below display the specific findings, which include the following: 


=" AOincreased the probability of being employed following the second quarter of enrollment, 
but the size of the impact is inconsistent and not always statistically different from zero. 
Following an initial decrease in employment, as may be expected while students are still 
engaged in education programming, students experienced an increase in employment that 
peaked at 8.0 percentage points in the 12th quarter following enrollment. This represents as 
much as a 15 percent increase. Through the 10th quarter, the effects of AO on employment 
appear to be fading out before a marked increase in the 11th and 12th quarters, apparently 
partially because of lower comparison group employment in those quarters (as shown in figure 
11). By the 12th quarter, however, the analysis only retained about 48 percent of the original 
sample because of time censoring, meaning that the results reflect the experience of an early 
subset of the AO population. 


= AOvincreased the average quarterly earnings for AO students somewhat in the medium term, 
but the effects disappear over time. The earnings gains are significantly different from zero 
only in a few quarters. The largest gain is in the fifth quarter following enrollment, when AO 
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students experienced earnings $293 higher on average than the comparison group, a gain of 14 
percent. Earnings then taper off and become not statistically different from zero. Even inthe 
11th and 12th quarters, when employment outcomes were relatively large and positive, there 
were no significant impacts on earnings, suggesting that the additional employment was in jobs 
with wages below the comparison group average. Though earnings impacts were not 
statistically different from zero, students in AO and the comparison group experienced growth 
in earnings over time (figure 13), meaning that the trajectory of growth was very similar for AO 
students and non-AO students. 


FIGURE 10 


Employment Impacts of AO by Quarter after Enrollment, Illinois 
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Note: Filled circles indicate significant impacts (p < 0.10); empty circles indicate impacts that are not statistically different from 
zero. 


Figure 11 illustrates changes in the employment rate associated with AO relative to employment 
rates of the comparison group in each quarter. The AO employment net impacts rise in quarters 11 and 
12 largely because the comparison group’s employment rate declines while AO students retain 
consistent levels of employment at about 60 percent. 
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FIGURE 11 


AO Impact on Employment Rates in Illinois 
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Note: *** p < 0.01, ** p < 0.05, *p < 0.10. 


Figure 12 graphs the net earnings impacts and figure 13 shows the earnings gains relative to the 
earnings levels of the comparison group. The net earnings impacts are modestly positive in quarters 4 
and 5, decline in quarter 6, are positive but not significantly different from zero in quarters 7 through 9, 
and then become negative but not significantly different from zero over quarters 10 through 12. The 
results indicate that the gains in credentials achieved by AO students did not materialize into robust 
increases in earnings. 
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FIGURE 12 


Earnings Impacts of AO by Quarter after Enrollment, Illinois 
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Note: Filled circles indicate significant impacts (p < 0.10); empty circles indicate impacts that are not statistically different from 
zero. 


The results in figure 13 highlight the similarity of earnings trends for the AO and comparison 
groups. For both groups, earnings increased substantially over the 12 quarters, more than doubling 
from about $1,300 per quarter to about $2,800 in the last quarters. These data illustrate the 
importance of including a comparison group. The absolute gains experienced by the AO group are large, 
but so were the absolute gains of the comparison group. By the sixth quarter after enrollment, AO 
participants experienced no earnings advantage over the non-AO workforce. 
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FIGURE 13 


AO Impact on Quarterly Earnings in Illinois 
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*** 9 < 0.01, ** p < 0.05, *p<0.10 


Kansas Impact Estimates 


Kansas Education Impacts 


The impact estimates for AO in Kansas cover the cohorts of AO participants enrolled from the 
beginning of the spring 2012 semester through the end of the fall 2014 semester. Data on educational 
activities for this report cover the period through the end of the spring 2015 semester, following 
students for 2 to 10 semesters after AO enrollment. The impact results for Kansas are presented in 
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table 6. 


The results vary substantially by recruitment source. Specifically, 


= AOincreased the number of credentials earned for all participants by 0.6 credentials on 
average, a 40 percent increase over the number of credentials earned by the comparison group. 
The average effects on credentials were lower among adult education students (0.1 credentials 
and not significant) than among CTE students, who experienced an increase of 0.8 credentials, 
representing a 57-percent increase. The adult education comparison group had roughly the 
same credential attainment as the CTE comparison group (1.5 and 1.4 credentials on average, 
respectively), so the gain relative to the comparison group was also lower for the adult 
education population. Although the comparable performance of adult education and CTE 
students in the comparison group may appear surprising, note that the adult education 
students who enrolled in AO are not typical in their pursuit of college relative to other adult 
education students, who may never enroll in a college level course. The CTE students in this 
study are college students with unusually low basic skills scores. Therefore, each group is not 
necessarily representative of the larger populations of adult education students and CTE 
students within the state. 


= AOincreased the probability that AO students earned any credential by 12.8 percentage 
points for the full sample, about a 19 percent increase over the probability of earning a 
credential in the comparison group. Again, the impact on AO students recruited from CTE was 
larger and more positive than the impact on students recruited from adult education. AO 
actually reduced the probability that AO students recruited from adult education earned any 
credentials by 0.7 percentage points, although this reduction is less than a 1 percent decrease. 
AO increased the probability of earning any credential by 20.4 percentage points for students 
recruited from CTE, a 30 percent increase. 


= AOreduced the average number of credits earned for the full sample by 3.5 credits. AO 
reduced the average number of credits earned by a substantial 10.2 credits for participants 
from adult education but only by 1.0 credit for participants that came from CTE. As in Illinois, 
the reduction in the number of credits earned might reflect acceleration, or students 
proceeding through courses more efficiently to achieve meaningful credentials. These data also 
align with an internal study of Kansas’s AO program, which found that the AO group did better 
in credential attainment and technical credits earned but that they earned fewer nontechnical 
credits. The internal study explained the effects by pointing to more directed advising into the 
pathway, which led to less exploration and a more efficient use of resources.*# 


= AOreduced the probability that participants earned more than 12 credits by 11.5 percentage 
points for the full sample. The impact was a 30.3 percentage-point reduction for participants 
from adult education but only a 4.8 percentage-point reduction for participants from CTE. 
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TABLE 6 


Education Impact Results for Kansas, by Recruitment Source 


Comparison 
group mean AO group mean AO gain over 
outcome outcome AO impact comparison (%) 

All (N = 14,293) 
Number of credentials earned 1.5 2.1 0.6*** 40% 
Any credential earned 68.6% 81.4% 12.8 pp*** 19% 
Number of credits earned 26.0 22.2 “3:5°"* -13% 
Earned more than 12 credits 65.6% 53.6% -11.5 pp*** -18% 
Recruited from adult education (N = 918) 
Number of credentials earned 1.5 1.6 0.1 7% 
Any credential earned 71.5% 63.2% -0.7 pp** -1% 
Number of credits earned 25.8 15.1 -10.2*** -40% 
Earned more than 12 credits 64.9% 34.9% -30.3 pp*** -47% 
Recruited from CTE (N = 13,375) 
Number of credentials earned 1.4 2.3 0.8*** 57% 
Any credential earned 67.6% 88.2% 20.4 pp*** 30% 
Number of credits earned 26.1 24.8 -1.0*** -4% 
Earned more than 12 credits 65.8% 60.5% -4.8 pp*** -7% 


Note: pp = percentage points. All sample sizes include both AO and non-AO groups. Impacts may not equal the exact difference 
between the comparison and treatment means because of rounding and regression adjustment of the impact estimates. 
*** ) < 0.01, ** p< 0.05, * p< 0.10 


Kansas Employment and Earnings Impacts 


The impact estimates for quarterly employment and earnings outcomes for AO in Kansas cover the 
cohorts of AO participants enrolled from the beginning of the spring 2012 semester through the end of 
the fall 2014 semester, similar to the educational impact estimates. Students’ employment and earnings 
are tracked for up to eight quarters following AO enrollment (two years).*° The impact results for 
employment and earnings for Kansas for each quarter are presented in and figure 14, figure 15, figure 
16, and figure 17. Note that all dollar amount averages include students who earned $0 per quarter. 


Overall, AO in Kansas exerted positive impacts on both employment and earnings, but the results 
vary substantially by recruitment source: 


= AOincreased the probability of being employed, but only for the CTE-recruited students. The 
increase for CTE students ranged from 2.4 to 6.9 percentage points, depending on the quarter. 
This represents as much as an 11-percent increase. The impact on employment for CTE 
students wavered from quarter to quarter after enrollment, peaking in the seventh quarter. 
Notably, employment was not negatively affected in the short term (i.e., participants did not 
forgo any earnings) during the period of AO enrollment. For adult education students, AO is 
generally associated with negative employment effects, although these are not statistically 
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significant. For the full sample, AO increased the probability of being employed by between a 
less than one percentage-point increase and a 5.0 percentage-point increase, depending on the 
quarter. 


= AOincreased the average quarterly earnings for AO students but increased them more 
consistently for CTE students than for adult education students. The gain for CTE students 
ranged from $325 to $1,188 depending on the quarter. The earnings of the AO group recruited 
from CTE are almost 33 percent higher than those of the comparison group from CTE in the 
eighth quarter after enrollment, at their peak. The positive effect of AO on average quarterly 
earnings is only statistically significant in the second quarter after enrollment for adult 
education students. AO increased earnings by $346 in that quarter (a 15 percent increase). For 
the full sample, the impact on earnings was between $233 and $811, depending on the quarter. 


Figure 14 graphs the absolute employment impacts while figure 15 displays these impacts 
alongside the employment levels of the comparison groups. Statistically insignificant impacts are 
indicated by empty circles in figure 14. The impacts for the CTE group remained positive through the 
entire postenrollment period, meaning that AO significantly increased CTE students’ employment rates. 
By the sixth quarter after enrollment in AO, the full sample of AO students did not experience 
statistically significant employment effects. 


FIGURE 14 


Employment Impacts of AO by Quarter after Enrollment, Kansas 
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Figure 15 shows the employment rate of the comparison group in each quarter and impacts 
experienced by AO students overall and from each recruitment source. The comparison group’s 
employment rates grew over time, but so did the impact of AO on employment. The adult education 
comparison group had a particularly high employment rate, except in the eighth quarter (the only 
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quarter in which positive and significant AO impacts appeared), when the comparison group’s 
employment rate was somewhat lower. The research team and state administrators do not have a firm 
explanation of the strong labor-market showing of the adult education comparison group. One 
possibility is that Kansas had a relatively strong economy during this period. The comparison group in 
Kansas was a particularly motivated and low-barrier subset of the adult education population that 
apparently did well in the local labor-market conditions. 
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FIGURE 15 
AO Impact on Employment Rates for Kansas 
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Figure 16 graphs the net earnings impacts while figure 17 shows the impacts alongside the trends in 


earnings levels for the comparison group. As with employment, earnings impacts for AO students 
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recruited from CTE are consistently higher than for the full sample of AO students. Also like the 
employment impacts, earnings impacts for AO students recruited from CTE continued to increase, 
reaching a 33 percent gain in the 8th quarter after enrollment. These represent net impacts in that AO 
increased CTE student earnings relative to the comparison group, which provides the best estimate of 
what AO participant earnings would have been had AO students not participated in AO. Although the 
research team did not estimate impacts separately by pathway because pathway selection occurs after 
program enrollment, a higher share of AO students recruited from CTE were enrolled in manufacturing 
pathways compared with AO students recruited from adult education. That may suggest an increased 
effectiveness of AO in manufacturing pathways. 


FIGURE 16 


Earnings Impacts of AO by Quarter after Enrollment, Kansas 
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Although AO students from CTE experienced significantly higher earnings than their comparison 
group, adult education students’ earnings generally did not exceed their comparison group’s earnings. 
Notably, both the CTE comparison group’s earnings and the adult education comparison group 
increased substantially over time. With AO, the CTE group achieved a 77 percent earnings gain from 
the first quarter after enrollment to the last quarter, higher than the 51 percent increase achieved by 
the comparison group over the same period. 
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FIGURE 17 
AO Impact on Quarterly Earnings in Kansas 
Quarterly earnings Comparison group = AO impact 
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Note: Data include participants with zero earnings. 
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Overall, CTE students in Kansas’s AO program experienced positive gains in credential attainment 
even while taking fewer credits overall. The CTE group experienced positive employment outcomes and 
earnings gains because of AO. Although AO’s adult education group did not significantly increase 
credential attainment, over 70 percent of both the AO group and comparison group earned at least one 
credential. Credential attainment effects were quite large for CTE participants, with 88 percent of the 
AO group earning at least one credential compared with 68 percent of the CTE comparison group. The 
impact on credits earned was negative for both the adult education and CTE groups. Although the CTE 
treatment and comparison groups began at similar levels, the gains over time were significantly higher 
among AO participants than among members of the CTE comparison group. By the sixth through eighth 
quarters following enrollment, AO students drawn from CTE programs were averaging $920 more in 
earnings per quarter than their comparison group counterparts. This represents a very large gain, 
perhaps partially explained by the strong alignment between Kansas CTE programs and labor market 
demand, which was coordinated beginning in 2010 by the Kansas Board of Regents. 


Kentucky Impact Estimates 


Kentucky Education Impacts 


The impact estimates for AO in Kentucky cover the cohorts of AO participants enrolled from the 
beginning of the spring 2012 semester through the end of the fall 2014 semester. The analysis follows 
these cohorts through the end of the spring 2015 semester, allowing 2 to 10 semesters to track 
outcomes. The impact results for Kentucky appear in table 7 and are similar across recruitment sources. 
The key findings on educational outcomes for AO in Kentucky are as follows: 


= _AOincreased the number of credentials earned for all participants by 0.7 credentials on 
average, representing a 78-percent increase over the number of credentials earned by the 
comparison group (which were quite low). The impact levels were the nearly same for the adult 
education and developmental-education groups, but the magnitudes of the impacts relative to 
the comparison group varied, with a much larger relative impact for the adult education AO 
group. 


= AOincreased the probability that AO students earned any credential by 19.0 percentage 
points for the full sample, representing a 133-percent increase over the likelihood of earning 
any credential among the comparison group. The impact on AO students recruited from adult 
education (19.8 percentage points) was again similar to the impact on students recruited from 
developmental education (19.5 percentage points) in absolute size. However, adult education 
AO students’ gain was substantially larger in relative terms at 566 percent compared with a 
127 percent increase for the developmental education AO group; that disparity was caused by 
differences in the comparison group values. Note that the impact of AO on earning any 
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credential is lower for the analysis of all AO students than it is for either the adult education or 
the developmental-education analyses. Those results are accurate and reflect differences in the 
regression adjustment between the subsamples and the total samples, known to statisticians as 
Simpson’s paradox. 


= _AOincreased the average number of credits earned for the full sample by 4.3 credits, a 30 
percent increase. AO increased the number of credits earned by 6.3 credits on average for 
participants that came from adult education, a 73-percent increase, and by 4.4 credits on 
average for participants that came from developmental education, a 29-percent increase. 


= _AOincreased the probability that participants earned more than 12 credits by 14.9 
percentage points for the full sample (a 38 percent increase), 21.8 percentage points for 
participants from adult education (a 114-percent increase), and 14.9 percentage points for 
participants from developmental education (a 36 percent increase). Those are quite large 
impacts, particularly for the adult education population. 


TABLE 7 


Education Impact Results for Kentucky, by Recruitment Source 


Comparison 
group mean AO group mean AO gain over 
outcome outcome AO impact comparison (%) 

All (N = 20,150) 
Number of credentials earned 0.9 1.7 0.7*** 78% 
Any credential earned 14.3% 33.6% 19.0 pp*** 133% 
Number of credits earned 14.4 18.8 4.3*** 30% 
Earned more than 12 credits 39.3% 54.7% 14.9 pp*** 38% 
Recruited from Adult Education (N = 2,814) 
Number of credentials earned 0.3 1.0 0.7*** 233% 
Any credential earned 3.5% 22.1% 19.8 pp*** 566% 
Number of credits earned 8.6 14.1 6.3*** 73% 
Earned more than 12 credits 19.1% 39.3% 21.8 pp*** 114% 
Recruited from Developmental Education (N = 17,336) 
Number of credentials earned 1.0 1.7 0.8*** 80% 
Any credential earned 15.4% 34.7% 19.5 pp*** 127% 
Number of credits earned 15.0 19.3 4.4*** 29% 
Earned more than 12 credits 41.3% 56.2% 14.9 pp*** 36% 


Note: pp = percentage points. All sample sizes include both AO and non-AO groups. Impacts may not equal the exact difference 
between the comparison and treatment means because of rounding and regression adjustment of the impact estimates. 
** 9) < 0.01, ** p< 0.05, * p< 0.10 
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Kentucky Employment and Earnings Impacts 


The impact estimates for quarterly employment and earnings in Kentucky cover the cohorts of AO 
participants enrolled from the beginning of the spring 2012 semester through the end of the fall 2014 
semester. The data follow students’ labor-market outcomes for up to 12 quarters (three years) after 
enrollment. 


The impact results for employment and earnings for Kentucky for each quarter are presented in 
figure 18, figure 19, figure 20, and figure 21. Note that all dollar amount averages include students who 
earned $0 per quarter. 


The impacts of AO on employment rates were positive for both groups, though earnings effects 
varied. Unlike in Kansas, the AO adult education group in Kentucky experienced significant earnings 
gains, while no significant gains emerged for the group who was recruited from a college source. 
Specifics are as follows: 


= AOincreased the probability of being employed for all AO participants. The increase for 
developmental education AO students ranged from zero to 4.9 percentage points, witha 
consistent 2 to 4 percentage-point impact in most quarters except for a dip in the eleventh 
quarter and slight upticks in the third observed quarter and the final observed quarter. Unlike 
developmental education students, AO students recruited from adult education experienced 
reduced employment levels and thus forgone earnings (negative impacts) in the initial quarters 
after starting the program. In subsequent quarters, AO’s positive effects on employment for 
adult education students reached 10.0 percentage points in the seventh quarter following 
enrollment, became statistically insignificant in the eighth quarter, and then were significant 
and positive in the ninth quarter before becoming statistically insignificant again. The pattern 
of impacts for the full sample looked very similar to the developmental education students. 


= AOincreased the average quarterly earnings consistently for AO students from adult 
education after an initial reduction in earnings in the first three quarters. The gain for adult 
education students reached $855 in the ninth quarter after enrollment, a 43 percent gain. The 
earnings of the AO group recruited from adult education peaked in the ninth quarter, when 
they were 43 percent higher than the comparison group. After the 10th quarter, the size of the 
impact declined somewhat but remained positive and statistically significant (ending with a 31 
percent increase relative to the comparison group in the last quarter). For AO participants 
recruited from developmental education, the impact on earnings was negative or zero through 
the 10th quarter and generally statistically significant; the final quarter indicated a positive and 
significant increase of $249, an 8 percent increase relative to the comparison group. The 
impacts for the full AO sample closely mirrored the impacts for the developmental education 
AO students. 


Figure 18 shows the net employment impacts and figure 19 shows the impacts in the context of the 
employment rates of the comparison group. Statistically insignificant impacts are indicated by an empty 
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circle in figure 18. AO students from adult education experienced negative employment and earnings 
impacts in the initial quarters after enrollment. By the fourth quarter after enrollment, however, AO 
students from both adult education and developmental education had achieved positive employment 
impacts as aresult of AO, though only the adult education group experienced consistently positive 
impacts on earnings after the initial dip. Overall, the impacts for the adult education group were more 
consistent and positive than the impacts for the developmental education group. 


FIGURE 18 
Employment Impacts of AO by Quarter after Enrollment, Kentucky 
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Figure 19 shows the employment rate of the comparison group in each quarter and the impact of 
AO on students overall and from each recruitment source. The developmental education comparison 
group’s employment rates grew over time. AO students from developmental education experienced 
employment rates slightly higher than the comparison group rate for most of the observed period. 
Employment rates among the adult education comparison group grew only slightly. AO students from 
adult education initially saw some positive and significant impacts, but then employment rates moved 
closer to the comparison group rate. 
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FIGURE 19 
AO Impact on Employment Rates for Kentucky 
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Note: Statistical significance is indicated as asterisks where *** p < 0.01, ** p < 0.05, * p< 0.10. 
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Figure 20 shows the net earnings impacts while figure 21 presents AO earnings alongside the 
earnings of the respective comparison groups. The impact of AO on earnings was consistently positive 
and statistically significant for students from adult education. In Kansas, in contrast, adult education 
students did not experience positive earnings impacts. Among developmental education students, no 
significant positive impacts emerged until the final quarter. 


FIGURE 20 
Earnings Impacts of AO by Quarter after Enrollment, Kentucky 
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Note: Filled circles indicate significant impacts (p < 0.10); empty circles indicate impacts that are not statistically different from 
zero. 


Figure 21 shows that AO students from adult education experienced a significant increase in 
earnings over the comparison group in most quarters. The adult education comparison group’s earnings 
did not increase over time (actually decreasing slightly between the first and last observed quarter), 
while earnings for the AO students increased about 92 percent. Absent AO, these adult education 
students would have experienced no earnings growth, but with AO, they experienced substantial 
growth. The earnings of the developmental education AO group grew at approximately the same rate as 
the comparison group, pulling ahead slightly in the final two quarters. 
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FIGURE 21 


AO Impact on Quarterly Earnings in Kentucky (Includes Zero Earnings) 
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Note: Statistical significance is indicated as asterisks where *** p < 0.01, ** p < 0.05, * p< 0.10. 
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Overall, adult education and developmental-education students in Kentucky’s AO program 
experienced positive gains in credential attainment and credit taking. That means AO promoted 
retention and college attachment, but the evidence does not suggest acceleration as seen in Illinois and 
Kansas. Acceleration may still be occurring, but acceleration cannot be disentangled from increased 
persistence given the limitations of the data. 


Although the labor-market impacts of AO in Kentucky were not positive and significant for the 
entire pool of AO participants, the employment and earnings gains among the adult education group 
were positive, significant, and substantively large. This pattern diverged from the results in Kansas, 
where AO exerted positive earnings impacts on CTE students but not on adult education students. 
Those results suggest that AO improved employment prospects and wages, particularly for the smaller 
group of students from adult education. 


Louisiana Impact Estimates 


Louisiana Education Impacts 


The estimates for Louisiana capture AO impacts on adult education students only, because AO students 
were drawn from only one recruitment source. The impact estimates for AO in Louisiana cover the 
cohorts of AO participants enrolled from the beginning of the fall 2012 semester through the end of the 
summer 2015 semester. The data on educational activities for this report cover the period through the 
end of the spring 2016 semester, tracking students for 3 to 11 semesters after enrollment. The 
education impact results for Louisiana appear in table 8. Overall, AO substantially increased the 
number of credentials but reduced the number of academic credits and persistence beyond 12 credits. 
Specifics are as follows: 


= AOincreased the number of credentials earned for all participants by 0.8 credentials on 
average, a large increase over the number of credentials earned by the comparison group, 
which was essentially zero. 


= AOincreased the probability that AO students earned any credential by 19.9 percentage 
points, an over six-fold increase in the likelihood of earning a credential in the comparison 
group, which was very low. 


= AOreduced the average number of credits earned for the full sample by 3.6 credits. As in 
Illinois and Kansas, the reduction in the number of credits earned might reflect accelerated 
pathways toward credentials. 
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= AOreduced the probability that participants earned more than 12 credits by 12.2 percentage 
points. That may be the result of students leaving college for employment after obtaining 
marketable credentials. 


TABLE 8 


Education Impact Results for Louisiana 


Comparison AO gain over 
group mean AO group comparison 
Outcome outcome mean outcome AO impact (%) 
Number of credentials earned 0.0 0.9 0.8*** NA 
Any credential earned 3.2% 23.4% 19.9 pp*** 622% 
Number of credits earned 14.4 9.7 -3.6°** -25% 
Earned more than 12 credits 40.8% 4.5% -12.2 pp*** -30% 


Note: N = 3,455; NA = not available because estimate would require dividing by zero. The impact may not equal the exact difference 
between the comparison and treatment means because of rounding and regression adjustment of the impact estimates. 
*** y < 0.01, ** p< 0.05, *p<0.10 


Employment and Earnings Impacts 


The AO impact estimates for quarterly employment and earnings outcomes in Louisiana cover the 
cohorts of AO participants who enrolled at any time from the beginning of the fall 2012 semester to the 
end of the summer 2015 semester. As in the other states, employment and earnings data are available 
for more quarters for the earlier cohorts (up to 12 quarters, or three years) than for the later cohorts. 
The latest cohorts had at least three quarters of follow-up. Thus, the long-term estimates include fewer 
observations than the short-term estimates. 


The impact results for employment and earnings for Louisiana for each quarter appear in figure 22, 
figure 23, figure 24, and figure 25. Note that all dollar amount averages include students who earned $0 
per quarter. 


Overall, AO exerted some positive impacts on employment and earnings through the sixth quarter 
after enrollment but did not lead to higher employment and earnings for AO participants after that 
point. Specifics are as follows: 


= AOincreased the probability of being employed in the short term, but the gains eroded after 
the sixth quarter after enrollment. The gains peaked at 10.8 percentage points in the fifth 
quarter following enrollment, a 22 percent gain over the comparison group. By the seventh 
quarter after enrollment, AO appears to have exerted a negative effect on employment of 1.8 
percentage points. Unlike in other states, AO in Louisiana exerted no negative effect on 
employment in the first few quarters after enrollment. 
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= AOalso increased the average quarterly earnings for AO students in the short term, but the 
earnings gains eroded beginning in the seventh quarter after enrollment, leading to negative 
quarterly earnings impacts. During the first five quarters after enrollment, AO participants 
consistently earned more than the comparison group (or what they would have earned without 
AO). In the 6th through the 12th quarter after enrollment, however, earnings impacts on AO 
participants turned consistently negative. That is, AO participants earned less than they would 
have earned in the absence of AO. The reduction in quarterly earnings reached $633 in the 
12th quarter following enrollment. 


As shown in figure 22, the employment impacts for AO students in Louisiana grew steadily over the 
course of the first year after enrollment but subsequently collapsed, producing no long-term 
employment gains in the observed period. 


FIGURE 22 


Employment Impacts of AO by Quarter after Enrollment, Louisiana 


Note: Filed circles indicate significant impacts (p < 0.10); empty circles indicate impacts that are not statistically different from 
zero. 


Figure 23 shows the employment rate of the comparison group in each quarter and impact of AO on 
students. State administrators pointed out that the declines for each cohort of AO participants relative 
to the comparison groups (see the “Cohort Effects” section below) roughly align with the rollout of the 
statewide Train to Attain model (an initiative inspired by AO) in 2014 as well as related initiatives that 
provided I-BEST-like services to adult education students across the state. The comparison group may 
have benefitted from a concurrent treatment similar to AO. 
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FIGURE 23 


AO Impact on Employment Rates in Louisiana 
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Note: *** p< 0.01, ** p< 0.05, * p< 0.10. 


Figure 24 graphs the earnings impacts. As with employment, earnings impacts are positive for an 
initial period after enrollment before turning negative. But unlike employment, for which the impacts 
returned to zero by the end of the observed period, the earnings impacts remained negative through 
the end of the 12 quarters. 


FIGURE 24 


Earnings Impacts of AO by Quarter after Enrollment, Louisiana 
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Note: Filled circles indicate significant impacts (p < 0.10); empty circles indicate impacts that are not statistically different from 
zero. 
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Figure 25 shows the comparison group’s average earnings (including those with zero earnings) and 
the incremental impact on earnings experienced by AO students. The comparison group’s earnings 
grew over this period, and the AO group’s earnings did not keep pace. Such a pattern of earnings 
impacts is the opposite of what would be expected from human capital development programs. In the 
early periods around enrollment, participants may experience forgone earnings, but those early 
shortfalls are offset by long-term gains. It is hard to understand why the earnings of AO participants fell 
consistently short of those of the comparison group. Even if the comparison group was able to take 
advantage of a competing program, such an intervention would be expected to equalize earnings 
between AO and control groups, not place the AO participants at a disadvantage. 


FIGURE 25 


AO Impact on Quarterly Earnings in Louisiana (Includes Zero Earnings) 
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Note: *** p < 0.01, ** p< 0.05, *p < 0.10. 


AO students in Louisiana experienced positive gains in credential attainment but completed fewer 
academic credits. AO seemed to exert short-run benefits for participants in Louisiana, peaking in the 
fifth quarter at $710 (a 38 percent increase over the comparison group). However, the fade-out of 
employment and earnings gains and the losses in earnings suggest that AO students were not 
consistently better off than other adult education students who enrolled in for-credit college courses. 
That may be partially because of the changes in the state’s adult education system over the period; 
many of those changes were structural and rendered the adult education system very similar to AO, 
with a focus on career pathways and strong navigation through the Train to Attain model. Further, the 
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state sponsored several grant initiatives, including a statewide TAACCCT grant from the US 
Department of Labor and an HPOG grant at several colleges. Therefore, AO may not have been very 
different from the standard services that students in the comparison group received, and AO appears to 
have benefitted AO students in the labor market less than those standard services benefitted the 
comparison group members. Finally, macroeconomic changes caused by falling oil prices may have had 
some effect. 


Cohort Effects 


Because the grant period covers three program years, not all participants are observed for the same 
number of follow-up quarters after enrollment for the labor-market analyses. If the impact of AO is 
consistent across program years, this data limitation will have not biased the results, though earlier 
postenrollment quarters will have larger sample sizes than later post-employment quarters, allowing 
for more precise detection of statistical significance. But if AO exerts different impacts across program 
years (for example, if program content improves as AO matures), then the impact estimates reported in 
the previous sections are unlikely to represent the gains of late-entering cohorts. As a result, identifying 
the patterns of AO impacts by cohort is important for providing context to the overall results. 


Figure 26, figure 27, figure 28, and figure 29 present the results of separate analyses of the impact 
of AO on annual cohorts in Illinois, Kansas, Kentucky, and Louisiana for a subset of AO colleges that are 
included in the cost-benefit analysis. The cost-benefit analysis colleges were running AO programs for 
all three years of the grant period, making them the most appropriate subset of colleges for an analysis 
of cohort effects. Including other colleges that only operated the AO program in the second and third 
years of the grant could conflate variations in the impact of AO across cohorts with variations in the 
impact of AO across colleges. By focusing only on colleges operating in all three years, the cohort 
analyses hold variations across colleges constant. 


The cohort-specific impact estimates for Illinois and Louisiana demonstrate the importance of 
understanding the differences in impact estimates across cohorts. In both cases, the earnings effects of 
AO for participants enrolling in the first program year are generally low or even negative (a notable 
exception is the impact on the sixth quarter of earnings in Louisiana, which is relatively high, with a gain 
of over $600). In contrast, the impact of AO for participants enrolling during the second program year is 
much higher than the impact for those enrolling during the first program year in both Illinois and 
Louisiana. Those estimates suggest that some maturation of the AO program may have occurred 
between the first and second year, leading to higher earnings impacts in the second year. 


In Kansas, the initial impact estimates for participants enrolling in the first year and the second year 
are relatively high. This is not surprising in light of the strong impact estimates for the total sample in 
Kansas. There is no evidence of program maturation in Kansas based on the cohort-specific impact 
estimates. AO participants in Kansas enrolling in the second program year saw reductions in earnings 
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caused by AO in quarter six. It is not clear whether those negative earnings persist for this cohort in 
Kansas, because no employment or earnings data are available for them after the sixth quarter. 


FIGURE 26 
Cohort-Specific Earnings Impacts of AO, by Quarter after Enrollment for CBA Colleges in Illinois 
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Notes: Year 1 represents cohorts that enrolled in 2012, year 2 represents cohorts that enrolled in 2013, and year 3 represents 
cohorts that enrolled in 2014. CBA = cost-benefit analysis. 


FIGURE 27 
Cohort-Specific Earnings Impacts of AO, by Quarter after Enrollment for CBA Colleges in Kansas 
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Notes: Year 1 represents cohorts that enrolled in 2012, year 2 represents cohorts that enrolled in 2013, and year 3 represents 
cohorts that enrolled in 2014. CBA = cost-benefit analysis. 
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FIGURE 28 
Cohort-Specific Earnings Impacts of AO, by Quarter after Enrollment for CBA Colleges in Kentucky 
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Notes: Year 1 represents cohorts that enrolled in 2012, year 2 represents cohorts that enrolled in 2013, and year 3 represents 
cohorts that enrolled in 2014. CBA = cost-benefit analysis. 


FIGURE 29 


Cohort-Specific Earnings Impacts of AO, by Quarter after Enrollment for CBA Colleges in Louisiana 
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Notes: Year 1 represents cohorts that enrolled from fall 2012 through summer 2013, year 2 represents cohorts that enrolled 
from fall 2013 through summer 2014, and year 3 represents cohorts that enrolled in fall 2014 through summer 2015. CBA = cost- 
benefit analysis. 
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Discussion 


AO’s purpose was to raise the education, skills, and employment success for adults with 
low basic skills. It also aimed to change the view of underprepared adult learners within 
state and college systems. Many policymakers, administrators, and staff were very 
skeptical initially that adults with low skills, especially those without high school 
credentials, could be successful in college programs. These results show that 
underprepared adult learners can be successful in college, earning more college- 


awarded credentials in fewer credits than their counterparts. 


The labor-market outcomes of these additional credentials are mixed in the time frame studied 
here, but low-skilled CTE students in Kansas and adult education students in Kentucky show signs of 
meaningful and lasting gains. In integrated career pathways that offer multiple points at which students 
can enter and exit with credentials, students may be more likely to “stop out” of college to enter the 
labor market more than they would if they did not have a structured pathway. Thus, earnings gains may 
take longer to manifest, possibly explaining results where students experienced positive employment 
impacts but no or negative earnings impacts, such as among AO students in Illinois and the 
developmental education population in Kentucky. Following students’ outcomes for more quarters 
would reveal a clearer picture of their academic and professional trajectories to draw stronger 
conclusions about long-term outcomes. Nonetheless, the appearance of any labor-market gains is of 
interest because research on the flagship I-BEST program in Washington State did not detect labor- 
market impacts, even though students had very positive education impacts (Zeidenberg, Cho, and 
Jenkins 2010). 


As Anderson and colleagues (2016) found in the implementation research, connections to 
employers and workforce systems were initially weak in many states and colleges but developed over 
time. Kansas seems to have had the most direct alignment between their CTE programs and labor force 
needs resulting from a statewide program alignment effort from 2010 through 2015. Across all states, 
placement of students into work-based learning grew from 28 percent in the first year to 42 percent in 
the third year, suggesting growing employer connections over the implementation period. In addition, 
other aspects of the program, such as team teaching, matured as the initiative continued. The longer- 
term impacts in this report represent the experiences of the earliest cohorts, who did not experience 
the most developed version of AO programming. These early cohorts, who enrolled in 2012, also faced 
particular economic circumstances during the early years of the recovery from the 2007 to 2009 
recession that may have affected their trajectories. Rising oil prices, shifts in manufacturing jobs, and 
other economic developments in these states and localities created conditions that may have affected 
cohorts of students differently. Following more cohorts for more time would make it easier to 
disentangle the effects. 


DISCUSSION (63) 


Overall, the AO population is a subset of students that may not represent the broader adult 
education, CTE, and developmental-education populations in their states. For example, the Kansas 
adult education group appears to be particularly advantaged as measured by educational attainment, 
prior employment, NRS functioning levels, and the earnings of the comparison group. The unique 
characteristics of AO students may limit the generalizability of the findings across larger groups. 


Further, students in particular state and local contexts experienced an array of complementary and 
competing services and programs that may have changed the added value of AO to students in those 
contexts. For example, during the demonstration period, the state of Louisiana was increasingly focused 
on career pathways and career navigation in its community colleges through their Train to Attain 
initiative. Large federal TAACCCT and HPOG grants in multiple states (particularly Louisiana) would 
have also improved the opportunity of many comparison group members to participate in programs 
similar to AO. Even if AO provided real benefits to participants in Louisiana, if those benefits are not 
large relative to the performance of comparison students enrolled in similar programs, then the 
estimated added benefits of AO will be minimal. Still, it is difficult to explain the significant negative 
earnings effects in Louisiana in the 6th through 12th quarters after enrollment. 


AO shows fairly consistent evidence of success on students’ education outcomes, with evidence of 
acceleration, or more credentials in fewer credits. One would expect college-awarded credentials to 
lead to lasting labor-market gains, but the evidence here is inconsistent. In Illinois, for example, it 
appears that AO students start out ahead in the labor market but then the comparison group is able to 
catch up. One possible explanation is that acceleration allows AO students to make gains earlier, but it 
does not result in lasting improved outcomes, at least for the population served in Illinois. Overall, more 
observation time is likely necessary to draw confident conclusions about the effectiveness of AO on 
improving employment outcomes. However, the CTE population in Kansas and adult education 
population in Kentucky suggest some promising longer-term labor-market gains. 
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Notes 


10. 


11. 


Bureau of Labor Statistics, “Usual Weekly Earnings of Wage and Salary Workers First Quarter 2017,” news 
release, April 18, 2017, https://www.bls.gov/news.release/pdf/wkyeng.pdf. 


For more detail, see Anderson et al. (2014); JFF’s Breaking Through website, “Breaking Through,” accessed 
May 23, 2017, http://www.jff.org/initiatives/breaking-through; and the Washington State Board of 
Community and Technical College’s I-BEST website, “I-BEST,” accessed May 23, 2017, 
https://www.sbctc.edu/colleges-staff/programs-services/i-best/. 


The foundations that supported AO included the Bill & Melinda Gates Foundation, the Joyce Foundation, the 
W. K. Kellogg Foundation, the Kresge Foundation, the Open Society Foundations, the Arthur M. Blank Family 
Foundation, the Robert W. Woodruff Foundation, the Annie E. Casey Foundation, and the University of 
Phoenix Foundation. The national technical assistance providers included the National College Transition 
Network, the National Council for Workforce Education, and the State Board for Community and Technical 
Colleges in Washington State. The evaluation states were Illinois, Kansas, Kentucky, and Louisiana. The 
affiliate states were Arkansas, Georgia, and Mississippi. 


Because the analysis depends on matching techniques and not randomized controlled trials, the results are 
subject to possible errors or biases. However, the extensive data available for the matching methods give 
confidence that the estimates in the analysis are unlikely to be subject to serious bias. 


Authors’ tabulations using the Bureau of Labor statistics data, based on data from the Current Population 
Survey. See “Databases, Tables & Calculators by Subject,” Bureau of Labor Statistics, accessed May 23, 2017, 
www.bls.gov/data. 


Bureau of Labor Statistics, “Usual Weekly Earnings of Wage and Salary Workers First Quarter 2017,” news 
release, April 18, 2017, https://www.bls.gov/news.release/pdf/wkyeng.pdf. 


Author tabulations using the Bureau of Labor Statistics data, based on data from the Current Population 
Survey (See “Databases, Tables & Calculators by Subject,” Bureau of Labor Statistics, accessed May 23, 2017, 
www.bls.gov/data). Of the 50 occupational areas projected to grow the fastest between 2014 and 2024, 74 
percent will require at least some postsecondary education, whereas 44 percent of the remaining 819 
occupational areas will require at least some postsecondary education. 


AO was managed by JFF in partnership with the National College Transition Network, the National Council for 
Workforce Education, and the State Board for Community and Technical Colleges in Washington State. JFF 
contracted with the Urban Institute and its partners, the Aspen Institute and the George Washington 
University, to independently evaluate the initiative. 


Some colleges received exceptions from JFF to offer 10- or 11-credit-hour pathways; others offered much 
longer pathways. For consistency, this analysis utilizes the “tipping point” of 12 credits as one indicator of 
persistence. Data limitations prevented the use of another measure of persistence, continued enrollment 
beyond the student’s pathway. The most substantial constraint faced in identifying completion of a pathway as 
an outcome variable was that student pathways were not identified in the data for three of the four states. 


For more detail, see Anderson et al. (2014); JFF’s Breaking Through website, “Breaking Through,” accessed 
May 23, 2017, http://www.jff.org/initiatives/breaking-through; and the Washington State Board of 
Community and Technical College’s I-BEST website, “I-BEST,” accessed May 23, 2017, 
https://www.sbctc.edu/colleges-staff/programs-services/i-best/. 


The report on the first year of AO implementation (Anderson et al. 2014) summarizes the AO model in depth. 
Anderson et al. (2016) provides information and lessons on AO implementation throughout the initiative. 
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12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


20. 


Zeidenberg, Cho, and Jenkins (2010) used propensity score matching to evaluate I-BEST. Kansas’s 
independent evaluation of AO (Ginther and Oslund 2016) also used this analysis method. 


For more detail on the implementation findings, see Anderson et al. (2014), Anderson et al. (2015), and 
Anderson et al. (2016). 


North Carolina initially participated in AO but later left the initiative to focus on state-sponsored efforts to 
accelerate low-skilled students in their Basic Skills Plus program. Louisiana joined the initiative around the 
time of North Carolina’s departure. 


The numbers did grow in later years of implementation; the size of the entering cohort across all four states 
grew from 2,370 students in 2012 to 3,043 students in 2014 according to college survey results (though these 
numbers differ somewhat from what is reported in the administrative data, as described below). 


There are different ways to match treatment and comparison cases using the propensity score. One is called 
“nearest neighbor,” where each treatment group member is matched to the comparison group member with 
the closest propensity score. Nearest neighbor can also be modified so that instead of matching one-to-one, 
the matches are one-to-five or one-to-ten, where each treatment group member is matched to multiple 
members of the comparison group. Another matching method is called “kernel density weighting.” Instead of 
directly matching one-to-one or one-to-many, kernel weighting weights the members of the comparison group 
by the distance of their propensity score from each treatment case. Kernel weighting uses every member of 
the comparison group within a certain propensity score distance of each treatment case, but gives higher 
weight to the comparison cases that are closer to the treatment case and a lower weight to the cases that are 
farther away. The distance of allowable propensity scores is called the “caliper” and the relative weight placed 
on more distant cases is determined by the “bandwidth.” This approach makes use of the most information 
available from the comparison group rather than only using information from one or a few neighbors. The 
estimates presented in this report are from kernel weighting at a bandwidth of 0.8. Sensitivity analyses using a 
bandwidth of 0.2 did not produce substantively different results. Nearest neighbor results were also not 
substantively different in most cases. 


The main drawback of propensity score matching is that it does not match people on unobservable 
characteristics such as motivation or intrinsic ability (though the inclusion of test scores may capture some 
aspects of ability). Therefore, it can only be used to draw unbiased causal inferences when all characteristics of 
the treatment and comparison groups that determine program participation are observed in the dataset and 
there are no unobserved determinants of program enrollment. This assumption cannot be directly tested, but 
it is typically justified by careful selection of the comparison group and a well-specified matching strategy. It 
does not provide the standard of evidence of an experiment, which is designed to hold both observable and 
unobservable characteristics constant across treatment and comparison groups. 


This approach builds on the analysis used in the I-BEST evaluation study (Jenkins et al. 2009) but enhances the 
I-BEST approach by including additional data. In particular, the expanded model used in this study controls for 
additional student characteristics in order to improve the match between AO and non-AO individuals and thus 
provide more reliable evidence of AO impacts on educational outcomes. Critically, the expanded model 
includes test scores, which were an important determinant of selection into AO and were not included in the I- 
BEST evaluation (Jenkins et al. 2009). The expanded model also includes information on local unemployment 
rates, program of study, and earning and employment history. For brevity, only the expanded model results 
appear in the body of this report. Results from a statistical model aligned with the I-BEST specification from 
Jenkins et al. (2009) are available from this report’s authors upon request. 


The Bureau of Labor Statistics Handbook reports that in 1997, 92.5 percent of the wage and salary component 
of national income is accounted for in the unemployment insurance records (see Bureau of Labor Statistics 
[1997]). This has increased over time, particularly following changes to federal regulations in 2014 that caused 
unemployment insurance to cover a wider range of earnings. 


Earnings were adjusted for inflation in all states except Kentucky, where it was impractical to do so given the 
structure of the data analysis. In Kentucky, the state could not release deidentified earnings records to 
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nonstate entities, so the research team wrote analysis code for the state to run on their data. It was 
exceedingly complex to write inflation adjustment code without having access to the structure of the 
underlying earnings records. However, inflation was relatively low, at less than 2 percent in 2010-16, and 
would not substantively change the Kentucky results. 


Personally identifiable information includes name, address, Social Security number, college ID, and date of 
birth. 


Limiting the population studied to students who took a for-credit college course also excludes from the 
comparison group adult education and developmental-education students who did not enroll in a for-credit 
college course. Jenkins et al. (2009) used this approach by comparing I-BEST students recruited from the adult 
education population with adult education students who were not in |-BEST but who enrolled in at least one 
college workforce course. In addition, the research team made the judgment that college course takers are 
most comparable to the AO participants. Broadening the comparison group to any adult education student, 
regardless of college enrollment, would likely produce an overestimate of the impact of AO, because AO 
participants are generally different from the broader adult education population in their higher (but 
unobserved) interest in and cultivation of occupational skills. 


At the same time, if many AO students who came from adult education would not have enrolled in college 
if not for AO, then comparing them to adult education students who accessed college without AO might end up 
comparing AO students to a set of non-AO students who are particularly motivated and had relatively low 
barriers to college access. Such an approach may lead to underestimates of the impacts of AO. The section 
“Adjustment for Comparison Group’s College Enrollment” describes one way that the research team 
attempted to reconcile this selection effect. 


The research team made the decision to look only at the first four courses because it roughly aligns with the 
12-credit AO pathway, assuming each course is three credits on average. The decision to designate courses as 
AO if they had at least four AO students in them was a somewhat arbitrary but seemingly reasonable cut-off to 
determine which courses were “AO courses.” 


States had the following ELL AO cases removed after all other sample cuts: 64 in Illinois, 6 in Kansas, 10 in 
Kentucky, and 3 in Louisiana. 


See, for example, Wilson (2014), which discusses how English-language learners generally have very different 
characteristics and motivations than low-skilled English-proficient students. 


Because Illinois has a very large adult education population, matching all potential comparison group members 
to state earnings records was not cost effective. To remain within the state’s budget, the research team 
conducted a preliminary propensity score match to narrow down the pool for which the state would pull UI 
earnings records. The preliminary match included all variables except for those available through the earnings 
records. The research team conducted a nearest neighbor match (20 neighbors, with replacement) to identify 
cases in the comparison group that were the most similar to the AO treatment group. All treatment cases were 
retained for the earnings match. The state matched this smaller comparison pool as well as all of the treatment 
cases with state earnings records, and the research team put the resulting sample through the full propensity 
score match utilizing a kernel density approach that included preprogram employment and earnings variables. 


The reason for limiting the sample to those with valid Social Security numbers was that valid Social Security 
numbers are required for linking individuals to state earnings records. The earnings records provide critical 
information that are used here for matching the AO and comparison groups and used in later analyses to 
estimate impacts on employment and earnings. Because the research team did not have access to students’ 
Social Security numbers, the states flagged those students who did not have a valid Social Security number so 
they could be dropped from the analysis. 


For example, according to state administrators, about 40 percent of adult education students in Kansas who 
complete their HSE transition to postsecondary education, but that is only about 6-7 percent of the overall 
adult education population. 
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See more detail about recruitment strategies by state in the chapter “Description of AO Students” and in 
Anderson et al. (2015). 


The credentials included in this report are only credentials awarded by the college, not credentials awarded by 
third parties. College credentials are often certificates of completion. These credentials are awarded and 
recorded by the college and are often occupationally related per the career pathways created by the colleges. 
However, the nature of the credentials varies widely across pathways, and there is no measure of the “value” 
of these credentials in the workplace (see Anderson et al. 2015). In addition, because the data come from the 
college system, they do not include third-party credentials such as state licenses and 
professional/occupational certifications. Jobs for the Future required the colleges and states to collect student 
information on third-party credential attainment for AO students later in 2012. Collecting these data is 
challenging as the data are self-reported by students and the quality and completeness may be suspect. In 
addition, states do not collect these data for the comparison group so they cannot be included in the impact 
analysis. 


As aresult of these limitations, estimates of the impact of AO on the number of credentials could 
understate the impact of AO on all credentials. Note however that the inability to observe non-college 
credentials does not necessarily mean that the impact estimate is an underestimate. If comparison group 
members earned more noncollege credentials than AO students, then the impact estimates presented here 
could overstate the impact on the number of credentials earned. The impact on noncollege credentials cannot 
be identified without data on noncollege credentials. 


Credentials offered in AO pathways were not always the same as those offered in non-AO programs. 
States and colleges faced clear incentives to increase the number of credentials offered in a pathway to meet 
the ambitious expectation from JFF that states award 3,600 credentials by the end of the three-year grant 
period. For this reason, the analysis measures both “any credential earned” and “number of credentials 
earned.” AO’s implicit incentive to increase credentials offered differs from I-BEST, because Washington State 
required that |-BEST programs have the same structure as similar non-I-BEST programs. 


The Louisiana data did not include noncredit academic activity. 


In fiscal year 2012 (though June 30, 2012), the Illinois adult education database, which was used in this 
analysis, only captured data on “gateway” courses to liberal arts programs. These are courses such as 
prerequisite math and writing courses. Although this gap should not bias the results because it applies to both 
the AO group and the comparison group, it may deflate the total number of courses taken. Data for all courses 
taken in Illinois were available after July 1, 2013. 


In all states, the treatment and comparison groups were rematched for each quarter of postprogram 
outcomes. 


The students profiled are in the sample of AO students included in the impact estimates before matching. This 
means that students who were excluded from the analysis because they did not take a credit-bearing course, 
did not have a valid Social Security number, or were English-language learners were also excluded from the 
profile. For simplicity, this chapter refers to this population as “AO students,” is the students discussed in this 
chapter are a subset of all students served. 


These are the percentages for the matched treatment population. Of the overall Kansas AO population, 30 
percent were from adult education and 70 percent were from CTE, according to the administrative records. 


Students who tested above the AO eligibility threshold and were in classes with AO students are not included 
in the analysis and are excluded from the comparison group. 


Asmall portion of students categorized as CTE came from developmental education programs or came to 
the college from external sources. Students were categorized as adult education or CTE per the following 
decision rule described in the text. Information from the implementation reports and unpublished analyses 
provided by the state roughly match the share of students categorized as CTE for the purposes of the analysis. 
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The research team flagged a Kentucky student as a developmental education student if they earned a 
traditional high school diploma before or in the year of AO entry (allowing for a traditional high-school-to- 
college trajectory) or if they earned an HSE at least two semesters before AO enrollment. Students who 
earned an HSE inthe semester before AO entry were considered recruited from adult education because adult 
education students were often encouraged to complete their HSE immediately before or very shortly after AO 
enrollment in order to qualify for federal financial aid. 


Identifying AO students and their pathways for the impact study differed from the how they were identified 
for the implementation study. For the implementation study, enrollment counts in AO overall and by pathway 
identifications were provided by AO college staff who completed the annual AO college surveys. For the 
impact study, enrollment in AO was determined by an AO flag in the data, and the assignment of pathways was 
determined empirically by examining the Classification of Instructional Programs (CIP) codes of the first four 
courses of enrollment after AO entry and any credentials earned in the first two semesters after enrollment. 
Therefore, enrollment numbers, particularly by pathway, may differ in this report relative to the earlier AO 
implementation reports (Anderson et al. 2014, Anderson et al. 2015, and Anderson et al. 2016). Sometimes it 
was not possible to discern a pathway because a student only took for-credit basic skills courses or courses in 
other areas that made it difficult to determine their substantive occupational area. 


Basic skill levels for students in specific pathways are not provided here, but they generally followed the same 
pattern of NRS functioning levels of the state as a whole. 


State administrators noted that they focused attention on students with intermediate NRS functioning levels 
because those students have historically been hardest to serve 


This type of separation may be possible if there existed an indicator of a student’s occupational interests at 
baseline (that is, before enrollment). However, because the research team determined occupational area 
based on actual course-taking and credentialing activity, the measure of occupation is endogenous to 
participation in AO and therefore not a useful covariate to segment the sample for impact estimation. 


Note that not all credentials are equal. A credential can be anything from a short-term certificate resulting 
from one or two courses through a degree. AO’s performance measures treated credentials more or less 
equally (for example, setting a goal of 3,600 credentials per state) and this analysis continues this approach. In 
future research, it may be valuable to distinguish types of credentials earned, but that level of analysis was 
beyond the purview of this effort. 


See, for example, Attewell et al. (2007). 


See, for example, Hotz, Imbens, and Klerman (2006); and Schochet, Burghardt, and McConnell, 2008. The 
results of the cost-benefit analysis will be strongly influenced by the trajectory of impacts (whether they 
continue to decline or level off). 


See Ginther and Oslund (2016). 


Because of the timing of data delivery, Kansas was only able to track students for 8 quarters while other states 
followed them for up to 12 quarters. 
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